Changelog

1.13.0 (2025-06-05)

Features (12 changes)

• webhook triggering while crawling pages (merge request)

• docker-compose based setup (merge request)

• document tags (merge request)

• search result CSV export (merge request)

• minimize database queries when workers are idle (merge request)

• remove the word stats button (merge request)

• use the script field of Crawl Policy to overwrite document’s data (merge request)

• set default policy to Crawl on depth (merge request)

• ask for confirmation when quitting a modified form (merge request)

• enable/disable search shortcuts (merge request)

• add an option to force reprocessing pages when no change is detected (merge request)

• run crawlers using half of the available CPU cores by default (merge request)

Bug fixes (2 changes)

• fix homepage links pointing to archive, when the should point to the source site (merge request)

• fix handling meta elements redirects with Request (merge request)

1.12.4 (2025-03-14)

Bug fixes (1 change)

• fix some parallel download failures using Firefox or Chromium (merge request)

1.12.3 (2025-02-24)

Bug fixes (1 change)

• fix fake_useragent in the pypi package (merge request)

1.12.2 (2025-02-02)

Bug fixes (1 change)

• fix documentation build

1.12.1 (2025-02-04)

Bug fixes (3 changes)

• fix counting doc with no language in the total count in analytics (merge request)

• fix displaying the 24h analytics graphs (merge request)

• fix docker build status when no upgrade is available

1.12.0 (2025-01-30)

Features (12 changes)

• enable browser detection by default (merge request)

• disable recursion in the default Crawl Policy (merge request)

• store the modification date of documents (merge request)

• display preview of images document in search results (merge request)

• add an orphan filter to the Document admin (merge request)

• batch URLs queueing (merge request)

• docker image daily upgrade (merge request)

• crawl status and crawlers info pages split (merge request)

• crawl all mimetypes by default (merge request)

• add a mimetype charts in the analytics page (merge request)

• upgrade geckodriver to version 0.35 (merge request)

• customizable “show on homepage” value when crawling a new URL (merge request)

Bug fixes (5 changes)

• fix loading CSS stylesheets that have rel=”preload” attribute (merge request)

• fix handling @import CSS directive (merge request)

• fix cropped pages screenshots (merge request)

• fix the unit of the last year indexing speed (merge request)

• fix path to admin’s images (merge request)

1.11.1 (2024-12-26)

Bug fixes (3 changes)

• avoid an exception when showing a cached page being crawled (merge request)

• prevent execution of scripts in cached pages (merge request)

• fix saving some CSS stylesheets (merge request)

1.11.0 (2024-12-09)

Features (6 changes)

• store file downloads (merge request)

• fake-useragent integration (merge request)

• home page optimization (merge request)

• multi-line url regexp in Crawl Policies (merge request)

• linkpreview integration (merge request)

• Netscape formatted cookies import (ticket)

Bug fixes (1 change)

• fix loading cookies for root url of domains

1.10.1 (2024-09-16)

Features (1 change)

• allow setting configurations option via environment variables (merge request)

Bug fixes (3 changes)

• fix running Chromium (merge request)

• fix Chromium file downloads (merge request)

• pin a Selenium version < 4.9 to prevent Selenium Manager to trigger (merge request / ticket)

1.10.0 (2024-07-04)

Features (5 changes)

• crawl policies enable/disable, copy (merge request)

• document hiding (merge request / ticket)

• UI dark theme (merge request)

• Statistics page rework (merge request)

• Rest API (merge request)

1.9.0 (2024-03-10)

Features (2 changes)

• ignore robots.txt file when it redirects in a loop (merge request)

• CLI command to mass delete documents (merge request)

Bug fixes (3 changes)

• fixes for long upgrade time and slow crawler startup (merge request)

• prevent user creation of Domain settings (merge request)

• fix browser mode detection (merge request)

1.8.2 (2024-01-23)

Bug fixes (2 changes)

• fix thumbnails deletion when deleting a document (merge request)

• pin working Selenium version (merge request)

1.8.1 (2024-01-14)

Bug fixes (2 changes)

• fix skipping pages with mimetype to be skipped (merge request)

• work-around to libmagic misidentifying some HTML page as application/javascript (merge request)

1.8.0 (2024-01-10)

Features (4 changes)

• HTML pages download with Requests (merge request)

• online status detection to optionally search locally when offline (merge request)

• improve page changes detection (merge request)

• add a “starts with” option to excluded urls (merge request)

Bug fixes (1 change)

• avoid recrawling documents after a redirect (merge request)

1.7.0 (2023-11-10)

Features (4 changes)

• navigation improvements (merge request)

• display the search bar in cached pages (merge request)

• set in the Crawl policy when to remove navigation related elements (merge request)

• save dynamic CSS in HTML snapshots (merge request)

Bug fixes (4 changes)

• fix font size on Chrome (merge request)

• fix downloading css assets containings special characters (merge request)

• avoid an infinite loops in particular conditions when a page is self-referencing (merge request)

• avoid timing issues during authentication (merge request)

1.6.0 (2023-10-14)

Features (2 changes)

• crawler Firefox 🦊 support (merge request)

• Atom and RSS feeds parsing (merge request)

1.5.0 (2023-08-29)

Bug fixes (5 changes)

• remove html comments from documents (merge request)

• handle invalid links correctly (merge request)

• fix result heights when thumbnail is enabled (merge request)

• fix an exception on links with unknown schemes (merge request)

• fix handling element in html snapshots (merge request)

1.4.0 (2023-08-24)

Features (1 change)

• create pages thumbnails (merge request)

Bug fixes (2 changes)

• add missing noreferrer flag to the unknown url page (merge request)

• avoid a race condition in HTML assets handling (merge request)

1.3.2 (2023-08-22)

Bug fixes (2 changes)

• fix handling some malformed urls

• fix the docker build

1.3.1 (2023-08-21)

Bug fixes (1 change)

• fix running with debug disabled (merge request)

1.3.0 (2023-08-21)

Features (4 changes)

• no referrer and open in new tab options for external links (merge request)

• HTML content snapshot (merge request)

• foldable header in screenshots view (merge request)

• Document admin page improvement (merge request)

Bug fixes (3 changes)

• update extern links if the target is indexed again afterward (merge request)

• fix sameSite cookie option formatting

• force screenshot refresh when they changed (merge request)

1.2.0 (2023-07-14)

Features (2 changes)

• crawling status improvements (merge request)

• navigation elements removal (merge request)

Bug fixes (1 change)

• fix screenshots resize (merge request)

1.1.2 (2023-06-30)

Bug fixes (1 change)

• fix database configuration documentation

1.1.1 (2023-06-29)

Bug fixes (1 change)

• misc Bookworm, Docker and Selenium fixes (merge request)

1.1.0 (2023-06-18)

Features (3 changes)

• Debian Bookworm support (merge request)

• browsable archive on homepage view (merge request)

• automatically follow redirection in cached pages (merge request)

Bug fixes (2 changes)

• fix resolving links when the header has a tag (merge request)

• scroll all elements when taking screenshots (merge request)

1.0.1 (2023-05-01)

Features (1 change)

• doc improvements

Bug fixes (1 change)

• build the Docker release using pypi packages