Extractors failing with PUID=PGID=33 #1366

alexschlueter · 2024-02-28T02:13:08Z

alexschlueter
Feb 28, 2024

Hi, I'm trying to run archivebox/archivebox:dev in docker compose, but I would like to set PUID and PGID environment variables to 33 (because I eventually want to use a folder owned by www-data on an SMB mount for the archives). Even with the data folder mounted as a named docker volume (so no SMB weirdness involved yet), multiple extractors are failing. If I don't set PUID and PGID, all extractors are working. How can I fix this?

0.7.3
ArchiveBox v0.7.3+editable COMMIT_HASH=fa24136 BUILD_TIME=2024-02-18 13:49:58 1708264198
IN_DOCKER=True IN_QEMU=False ARCH=x86_64 OS=Linux PLATFORM=Linux-5.10.0-28-amd64-x86_64-with-glibc2.36 PYTHON=Cpython
FS_ATOMIC=True FS_REMOTE=True FS_USER=33:33 FS_PERMS=644
DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND=ripgrep LDAP=False
[i] Dependency versions:
 √  PYTHON_BINARY         v3.11.8         valid     /usr/local/bin/python3.11
 √  SQLITE_BINARY         v2.6.0          valid     /usr/local/lib/python3.11/sqlite3/dbapi2.py
 √  DJANGO_BINARY         v3.1.14         valid     /usr/local/lib/python3.11/site-packages/django/__init__.py
 √  ARCHIVEBOX_BINARY     v0.7.3          valid     /usr/local/bin/archivebox
 √  CURL_BINARY           v8.5.0          valid     /usr/bin/curl
 √  WGET_BINARY           v1.21.3         valid     /usr/bin/wget
 √  NODE_BINARY           v20.11.1        valid     /usr/bin/node
 √  SINGLEFILE_BINARY     v1.1.46         valid     /app/node_modules/single-file-cli/single-file
 √  READABILITY_BINARY    v0.0.11         valid     /app/node_modules/readability-extractor/readability-extractor
 √  MERCURY_BINARY        v1.0.0          valid     /app/node_modules/@postlight/parser/cli.js
 √  GIT_BINARY            v2.39.2         valid     /usr/bin/git
 √  YOUTUBEDL_BINARY      v2023.12.30     valid     /usr/local/bin/yt-dlp
 √  CHROME_BINARY         v121.0.6167.57  valid     /usr/bin/chromium-browser
 √  RIPGREP_BINARY        v13.0.0         valid     /usr/bin/rg
[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /app/archivebox
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates
 -  CUSTOM_TEMPLATES_DIR  -               disabled  None
[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled  None
 -  COOKIES_FILE          -               disabled  None
[i] Data locations:
 √  OUTPUT_DIR            5 files @       valid     /data
 √  SOURCES_DIR           1 files         valid     ./sources
 √  LOGS_DIR              1 files         valid     ./logs
 √  ARCHIVE_DIR           1 files         valid     ./archive
 √  CONFIG_FILE           81.0 Bytes      valid     ./ArchiveBox.conf
 √  SQL_INDEX             208.0 KB        valid     ./index.sqlite3

$ > docker compose run --rm archivebox add https://example.com/
[i] [2024-02-28 02:09:19] ArchiveBox v0.7.3: archivebox add https://example.com/
    > /data
[+] [2024-02-28 02:09:21] Adding 1 links to index (crawl depth=0)...
    > Saved verbatim input to sources/1709086161-import.txt
    > Parsed 1 URLs from input (Generic TXT)
    > Found 1 new URLs not already in index
[*] [2024-02-28 02:09:21] Writing 1 links to main index...
    √ ./index.sqlite3
[*] [2024-02-28 02:09:21] Archiving 1/1 URLs from added set...
[▶] [2024-02-28 02:09:21] Starting archiving of 1 snapshots in index...
[+] [2024-02-28 02:09:21] "example.com"
    https://example.com/
    > ./archive/1709086161.608541
      > favicon
      > headers
      > singlefile
        Extractor failed:
             SingleFile was not able to archive the page
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            /app/node_modules/single-file-cli/single-file --browser-executable-path=chromium-browser "--browser-args=[\"--headless=new\", \"--no-sandbox\", \"--no-zygote\", \"--disable-dev-shm-usage\", \"--disable-software-rasterizer\", \"--run-all-compositor-stages-before-draw\", \"--hide-scrollbars\", \"--autoplay-policy=no-user-gesture-required\", \"--no-first-run\", \"--use-fake-ui-for-media-stream\", \"--use-fake-device-for-media-stream\", \"--disable-sync\", \"--simulate-outdated-no-au='Tue, 31 Dec 2099 23:59:59 GMT'\", \"--window-size=1440,2000\", \"--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.3 (+https://github.com/ArchiveBox/ArchiveBox/)\"]" "https://example.com/" singlefile.html
      > pdf
        Extractor failed:
             Failed to save PDF
            chrome_crashpad_handler: --database is required
            Try 'chrome_crashpad_handler --help' for more information.
            [143:143:0228/020923.342009:ERROR:socket.cc(120)] recvmsg: Connection reset by peer (104)
            [143:143:0228/020923.342267:FATAL:crashpad_linux.cc(195)] Check failed: client.StartHandler(handler_path, *database_path, metrics_path, url, annotations, arguments, false, false).
            #0 0x561630a9f932 base::debug::CollectStackTrace()
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            /usr/bin/chromium-browser --headless=new --no-sandbox --no-zygote --disable-dev-shm-usage --disable-software-rasterizer --run-all-compositor-stages-before-draw --hide-scrollbars --autoplay-policy=no-user-gesture-required --no-first-run --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --disable-sync "--simulate-outdated-no-au='Tue, 31 Dec 2099 23:59:59 GMT'" --window-size=1440,2000 "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.3 (+https://github.com/ArchiveBox/ArchiveBox/)" --print-to-pdf "https://example.com/"
      > screenshot
        Extractor failed:
             Failed to save screenshot
            chrome_crashpad_handler: --database is required
            Try 'chrome_crashpad_handler --help' for more information.
            [147:147:0228/020923.873534:FATAL:crashpad_linux.cc(195)] Check failed: client.StartHandler(handler_path, *database_path, metrics_path, url, annotations, arguments, false, false).
            #0 0x5650f477f932 base::debug::CollectStackTrace()
            #1 0x5650f476d573 base::debug::StackTrace::StackTrace()
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            /usr/bin/chromium-browser --headless=new --no-sandbox --no-zygote --disable-dev-shm-usage --disable-software-rasterizer --run-all-compositor-stages-before-draw --hide-scrollbars --autoplay-policy=no-user-gesture-required --no-first-run --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --disable-sync "--simulate-outdated-no-au='Tue, 31 Dec 2099 23:59:59 GMT'" --window-size=1440,2000 "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.3 (+https://github.com/ArchiveBox/ArchiveBox/)" --screenshot "https://example.com/"
      > dom
        Extractor failed:
             Failed to save DOM
            chrome_crashpad_handler: --database is required
            Try 'chrome_crashpad_handler --help' for more information.
            [151:151:0228/020924.330280:FATAL:crashpad_linux.cc(195)] Check failed: client.StartHandler(handler_path, *database_path, metrics_path, url, annotations, arguments, false, false).
            #0 0x555ab663d932 base::debug::CollectStackTrace()
            #1 0x555ab662b573 base::debug::StackTrace::StackTrace()
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            /usr/bin/chromium-browser --headless=new --no-sandbox --no-zygote --disable-dev-shm-usage --disable-software-rasterizer --run-all-compositor-stages-before-draw --hide-scrollbars --autoplay-policy=no-user-gesture-required --no-first-run --use-fake-ui-for-media-stream --use-fake-device-for-media-stream --disable-sync "--simulate-outdated-no-au='Tue, 31 Dec 2099 23:59:59 GMT'" --window-size=1440,2000 "--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.3 (+https://github.com/ArchiveBox/ArchiveBox/)" --dump-dom "https://example.com/"
      > wget
      > title
        Extractor failed:
             Unable to detect page title
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            curl --silent --location --compressed --max-time 60 --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.3 (+https://github.com/ArchiveBox/ArchiveBox/) curl/curl 8.5.0 (x86_64-pc-linux-gnu)" "https://example.com/"
      > readability
        Extractor failed:
             Readability could not find HTML to parse for article text
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            curl ./{dom,singlefile}.html
      > mercury
      > htmltotext
        Extractor failed:
             htmltotext could not find HTML to parse for article text
        Run to see full output:
          docker run -it -v $PWD/data:/data archivebox/archivebox /bin/bash
            cd /data/archive/1709086161.608541;
            "(internal) archivebox.extractors.htmltotext" ./{singlefile,dom}.html
      > media
        11 files (18.1 MB) in 0:00:08s
[√] [2024-02-28 02:09:30] Update of 1 pages complete (8.67 sec)
    - 0 links skipped
    - 1 links updated
    - 1 links had errors
    Hint: To manage your archive in a Web UI, run:
        archivebox server 0.0.0.0:8000

pirate · 2024-02-29T01:09:07Z

pirate
Feb 29, 2024
Maintainer

In general www-data is a very restricted user, I'm not surprised it's breaking things. All uids below 100 are sort of a special case because they often have pre-existing permissions restrictions on most linux systems, and setting archivebox to use that uid is basically inheriting all of the restrictions that uid already has.

I think uid=33 maps to an existing unprivileged www-data user inside docker, so everything is failing because that user doesn't have enough permissions to write to any of the system /var, etc. directories that it needs during archiving.

You could try leaving PUID as the default, but setting PGID=33? (or vice versa if that fails)

You can also experiment with creating a new group outside of docker, adding www-data to that group, and then passing in its PGID. This will allow the default archivebox (uid=911) user to run the archiving processes inside Docker (which will fix all the errors you're seeing), while also keeping the data dir owned by the group ID that gives your host www-data user access outside of docker.

4 replies

alexschlueter Feb 29, 2024
Author

Thanks for the suggestions! Unfortunately, both are not working:

With PUID unset, GUID=1003 (group with www-data), it crashes because you need to be owner in order to chmod:

archivebox-1  | [*] [2024-02-29 02:03:34] Archiving 1/1 URLs from added set...
archivebox-1  | Internal Server Error: /add/
archivebox-1  | Traceback (most recent call last):
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
archivebox-1  |     response = get_response(request)
archivebox-1  |                ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 181, in _get_response
archivebox-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
archivebox-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 70, in view
archivebox-1  |     return self.dispatch(request, *args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/utils/decorators.py", line 43, in _wrapper
archivebox-1  |     return bound_method(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
archivebox-1  |     return view_func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 109, in dispatch
archivebox-1  |     return super().dispatch(request, *args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 98, in dispatch
archivebox-1  |     return handler(request, *args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/edit.py", line 142, in post
archivebox-1  |     return self.form_valid(form)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/core/views.py", line 290, in form_valid
archivebox-1  |     add(**input_kwargs)
archivebox-1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox-1  |     return func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/main.py", line 693, in add
archivebox-1  |     archive_links(new_links, overwrite=False, **archive_kwargs)
archivebox-1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox-1  |     return func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/extractors/__init__.py", line 236, in archive_links
archivebox-1  |     archive_link(to_archive, overwrite=overwrite, methods=methods, out_dir=Path(link.link_dir))
archivebox-1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox-1  |     return func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/extractors/__init__.py", line 133, in archive_link
archivebox-1  |     write_link_details(link, out_dir=out_dir, skip_sql_index=False)
archivebox-1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox-1  |     return func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/index/__init__.py", line 335, in write_link_details
archivebox-1  |     write_json_link_details(link, out_dir=out_dir)
archivebox-1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox-1  |     return func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/index/json.py", line 99, in write_json_link_details
archivebox-1  |     atomic_write(str(path), link._asdict(extended=True))
archivebox-1  |   File "/app/archivebox/util.py", line 116, in typechecked_function
archivebox-1  |     return func(*args, **kwargs)
archivebox-1  |            ^^^^^^^^^^^^^^^^^^^^^
archivebox-1  |   File "/app/archivebox/system.py", line 108, in atomic_write
archivebox-1  |     os.chmod(path, int(OUTPUT_PERMISSIONS, base=8))
archivebox-1  | PermissionError: [Errno 1] Operation not permitted: '/data/archive/1709172214.318422/index.json'

With PUID=33, GUID unset, the extractors fail with the same messages as in my original post.

However, I found a different solution using this docker plugin for bindfs to rewrite the owner that docker sees: https://github.com/clecherbauer/docker-volume-bindfs
In docker-compose.yml:

volumes:
  archivebox-archive:
    driver: lebokus/bindfs:latest
    driver_opts:
      sourcePath: "${EXTERNAL_MOUNT_PARENT}/external-parent/external/archivebox"
      map: "33/911:@33/@911"

pirate Feb 29, 2024
Maintainer

With PUID unset, GUID=1003 (group with www-data), it crashes because you need to be owner in order to chmod

You might be able to fix this error by doing chown -R 911:1003 archive before you start Docker Compose.

alexschlueter Feb 29, 2024
Author

Unfortunately I can't change the owner of the folder because it is on an external SMB mount which has fixed owner and permissions.

pirate Feb 29, 2024
Maintainer

ah yeah that wont work if it's fixed to 33, you definitely need bindfs

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Extractors failing with PUID=PGID=33 #1366

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment · 4 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Uh oh!

Extractors failing with PUID=PGID=33 #1366

Uh oh!

Uh oh!

alexschlueter Feb 28, 2024

Replies: 1 comment · 4 replies

Uh oh!

Uh oh!

pirate Feb 29, 2024 Maintainer

Uh oh!

alexschlueter Feb 29, 2024 Author

Uh oh!

pirate Feb 29, 2024 Maintainer

Uh oh!

alexschlueter Feb 29, 2024 Author

Uh oh!

pirate Feb 29, 2024 Maintainer

alexschlueter
Feb 28, 2024

pirate
Feb 29, 2024
Maintainer

alexschlueter Feb 29, 2024
Author

pirate Feb 29, 2024
Maintainer

alexschlueter Feb 29, 2024
Author

pirate Feb 29, 2024
Maintainer