Structr 7.1.0
- New content extraction module replaces the text-search module: it reads over a thousand file types, including scans via OCR, and finds files by their content.
- Vector search, named job queues that run in parallel, Excel workbooks with several sheets and an optional second factor for the superuser.
- Pages reached through URL routes get their
currentobject back; newminify()function and a server log REST endpoint. - Safer defaults: HttpOnly session cookie, sandboxed delivery of untrusted files and a setup token for fresh installations.
- Read the Upgrade Notes first: content search is opt-in, the Debian package no longer installs Java, and mailboxes are no longer fetched automatically.
Structr 7.1 replaces the text-search module with a new content extraction module, adds vector search, named job queues, Excel workbooks with several sheets and a second factor for the superuser, and makes file delivery, the session cookie and transactions under load safer. Please read the Upgrade Notes before upgrading: several behaviour changes need a look at existing applications.
Several behaviour changes need a look at existing applications. Read the Upgrade Notes before upgrading a production instance.
Read the Upgrade NotesNew Features
Content Extraction
- The new content-extraction module replaces the text-search module, which is removed. It reads the text of over a thousand file types, including PDF, office and OpenDocument files, HTML, Markdown and e-mails, and reads scanned documents and images with OCR when Tesseract is installed. The text is stored with the file in chunks, with the page, the heading path and, with the option Include Embedded, the entries of archives and the attachments of e-mails.
- Extraction is configured per folder with the new Content Extraction dialog of the Files area and runs in the background, visible in the Job Queue. It covers uploads by every channel, replaced content, moved files, mounted folders and deployment imports. The search box of the Files area and the global search find files by their content.
- New
$.documentsfunctions work with the extracted text in scripts, among themfulltextSearch(),getExtractedText(),processContent()andstopWords(). With Structr Enterprise, the chunks can also be embedded with an LLM provider, which addsdocuments.vectorSearch()anddocuments.answer(), an answer with citations from the passages found.
Vector Search
Double[]attributes can have a vector index. The Vector Index dialog of a schema property sets the size of the vectors and the similarity; Structr creates the index and recreates it when they change, and refuses vectors that the index would leave out.searchVector(indexName, vector, limit)returns the closest objects the user can read. With Structr Enterprise,llmEmbed()makes the vectors with an embedding model, andcosineSimilarity()compares two of them. The new documentation chapter "Search & Retrieval" explains which search function to use for which purpose.
Named Job Queues
- The job queue is now a set of named queues: each runs one job at a time, and different queues run in parallel, so a long-running job no longer holds up every other scheduled job.
schedule()takes a trailing options object withqueue, for exampleschedule(fn, { queue: 'imports' }), and the queue of a job shows in the Job Queue area, the job list andjobInfo().schedule()now returns the id of the job, as documented.
Excel Workbooks With Several Sheets
toExcel()writes workbooks with several sheets, with a new call style; calls in the old style are reported and migrated by the migration service. Wrong attribute names produce a warning instead of an exception, andmaxCellLengthworks in StructrScript.
Second Factor for the Superuser
- The superuser can be asked for a code from an authenticator app (TOTP). Switch on
superuser.twofactor.enabled; the next login to the configuration interface shows a QR code to set up the app. Superuser requests over HTTP then need the current code in the newX-TwoFactor-Codeheader, one request per code (the code before the current one is accepted as well, and reusing a code is refused without counting as a failed attempt), and HTTP Basic authentication is refused for the superuser. Off by default, so existing installations log in as before.
Current Object and Partial Reloads with URL Routes
- A page reached through a URL route has a
currentobject again. Mark a route parameter as Current object in the URL routing dialog; without a marked one, the only Node parameter of a route is used. A parameter of another type is looked up as a UUID. Withcurrent, theidkeyword anddata-current-object-idwork on routed pages as well. - Partial reloads on a page reached through a URL route see the route's values: the reload sends the page's path, and Structr resolves the route again with the user's permissions.
Minifying Web Resources
- New function
minify(file [, outputPath ])($.minify()in JavaScript) writes a minified copy of a JavaScript, CSS, HTML, SVG, XML or JSON file from the file area and returns the new file. The type is detected from the content type or the extension; JavaScript is minified with terser and CSS with csso, both bundled and run in an isolated sandbox, HTML, SVG and XML lose comments and redundant whitespace, and JSON all whitespace. WithoutoutputPath,/js/app.jsis written to/js/app.min.js; calling it again overwrites the minified file.
Server Log Endpoint
- New REST endpoint
GET /structr/rest/_serverLogreturns the server log, so it can be collected by scripts and monitoring tools without the dashboard. It reads the log as entries, so an error comes back with its stack trace, and takeslines(default 50),level(a minimum level such asERROR),filter,logFileandsince: every response carries a byteoffset, and passing it assincereturns only what was logged after it, which keeps frequent polling cheap however large the log grows. The endpoint is restricted to admin users; a resource permission for anyone else is refused with 403.
Enhancements
Security
- The Admin UI works with
httpservice.cookies.httponly = true. It no longer reads theJSESSIONIDcookie: the websocket takes the session from the cookie the browser sends with the handshake, for a handshake from the same origin or from an origin listed explicitly inaccess.control.accepted.origins(a*there does not count). Clients that send a session id in their websocket messages, such as Structr Desktop, work as before. Until 7.0 the Admin UI showed a warning advising to turn HttpOnly off; nowhttpservice.cookies.httponlyis on by default. - A fresh installation asks for a one-time setup token before the initial configuration. Structr writes the token to the server log at every start until the setup is completed; automated installations preset it with
setup.token(STRUCTR_setup_tokenin Docker). - New setting
httpservice.force.https.permanentanswers the HTTPS redirect with 301 instead of 302. - Logging in to the token resource or the websocket with the superuser credentials no longer causes an exception that leaked information.
- The documentation describes the superuser protection wherever the superuser appears, and the Best Practices recommend the superuser's second factor and the HttpOnly cookie.
Scripting and Event Action Mapping
- In
onSavemethods, the modifications are available as the keyword$.modifications, in addition to$.methodParameters.modifications. - Inline notifications of event action mappings can access the result data of the action.
- The GraalVM statement count limit can be changed without a restart.
sort()and the sort predicate offind()accept several sort keys in any combination of names, orders and objects; see the Upgrade Notes for two changed calls.
Files and Storage
- The General tab of a file shows all its checksums, and the General tab of a folder has a quick switch for enabling checksums.
File.trustedandFolder.enabledChecksumsare part of a deployment. - Mounted storages: the per-mount sync settings (watch, scan interval, file and folder type) moved from the folder into the storage configuration, so they survive a deployment round trip. The flag
sync.deleteStaleis replaced by the policysync.onVanishedwithignore,deleteandreadonly, editable in the mount dialog; existing settings are migrated. Folder cases on storages that do not move folders as a whole, S3 among them, are handled correctly. - The search box of the Files area finds files by name as well as by content.
Deployment
pre-deploy.confandpost-deploy.confare part of the deployment data, can be edited in the Code area and are found by the global search and the code search.- The deployment export includes custom relationships of files and pages.
Messaging
- MQTT, Kafka and Pulsar clients deliver messages reliably, and XMPP clients reconnect. The documentation describes the delivery guarantees of each broker.
Database and Transactions
- A fresh database no longer fails to start with "insanely frequent schema changes": transactions terminated by schema changes are repeated with exponential backoff, also in authentication, page rendering and test cleanup.
- Retryable failed transactions are repeated instead of committed, which also prevents duplicate \*-to-one relationships under concurrency. Repeated transactions no longer duplicate the results of PATCH, POST and CSV requests, and callbacks after commit stay repeatable.
- Better prefetching, lower memory use and fewer queries in the Bolt driver.
Monitoring
- The HTTP access statistics on the dashboard survive a restart: they are saved every 5 minutes, and on shutdown, to
logs/http-access-statistics.jsonin the base path. The new settingapplication.stats.retention.days(default 30) limits how far back they reach, which also caps their memory use; until 7.0 they grew for as long as the instance ran. The file is not part of the database or a deployment export, so copy thelogsdirectory when moving an instance to another server and include it in backups. - The query histogram records timings again on every database driver, including the blocking Bolt transaction that is the default for Neo4j 4 and later, and keeps bounded totals per statement.
Configuration Interface
- The Letsencrypt section of the configuration interface has a Retrieve certificate button. It runs the
letsencryptmaintenance command with the saved settings against the production or staging server, optionally reloads the certificate in the running HttpService, and shows the result, so getting a first certificate no longer needs a user-defined method. - Cron expressions are checked in the configuration interface with the same rules as on the server, and the documentation explains the full syntax with examples.
Admin UI
- Documentation area: the navigation and the "On this page" columns are as wide as their content and can be collapsed; the article takes the rest of the window. Features of Structr Enterprise are marked with a panel and an Enterprise label.
- Scratchpad: a "Copy output" button, and the log can be resized.
- The global search shows a spinner while it runs and also searches deployment configuration scripts.
- The Monaco editor is updated to 0.55.1.
- The build number in the footer has eight characters and is shown as plain text.
Operations
- The Debian package's
stop,startandstatusscripts act only on the process in the PID file, so they no longer stop or detect other Structr instances on the same host;stopwaits for Structr to shut down. - The Docker image contains fonts. After a release, the Docker tags
MAJOR,MAJOR.MINORandlatestare moved to it.
Bug Fixes
login()no longer hangs forever when the same transaction wrote the user node before, and re-hashing a password on login no longer deadlocks.passwordAttemptsis incremented only once per failed attempt.- A principal is looked up per authentication key again, so a key declared on a subtype no longer locks everyone else out.
- Methods: fixed a deadlock in the flow module and the loop bodies of FlowForEach and FlowFork.
- Sorting: sorting by a path expression no longer breaks pagination, a wrong sort key is reported, remote collections sort correctly, and the in-memory driver keeps the remaining sort keys when both values are null.
- Queries: advanced find works with string array properties, queries for non-empty results work, and an exception in prefetching is fixed.
- Pages: no duplicate DOCTYPE, table fragments such as
tdare parsed correctly again, an element without a name no longer fails the request, and a create action no longer stores thecurrentObjectIdof the event on the new object. - Websocket: nested objects in JSON are resolved correctly.
- Jetty: no more errors from closed connections, and a script that reads the request after its HTTP exchange has completed no longer gets an exception or the headers of the next request.
- Schema: inherited views are cleaned up, a schema relationship is no longer renamed by unrelated changes, and an overridden
nameproperty that returns a non-string value no longer throws a ClassCastException. - The CSV import dialog loads stored configurations correctly, and the CSV import stores empty arrays correctly.
- The FTP server changes to the root on CDUP from a top-level folder.
- The Cypher console and
cypher()withrunInNewTransactionwork on the embedded database driver. - Mail: S/MIME-signed messages no longer make the fetch fail.
- Scheduled scripts no longer fail when the scheduling principal has no security context.
- The ODF exporters work again.
- Rebuilding the index for a single relationship type uses the given type.
- The configuration interface lists the available drivers of existing database connections correctly, and the setup wizard shows the form for the superuser credentials again.
Upgrade Notes
- Content search replaces the text-search module, and is opt-in. Files are only extracted, and found by their content, in folders with a content extraction configuration: after the upgrade no file is found by its content until an admin creates one in the Content Extraction dialog of the Files area and processes the existing files. With the default
application.migration.mode = apply, the old data is deleted on startup: theextractedContentandindexedproperties of all files,mountDoFulltextIndexingof all folders and theFile_extractedContent_fulltextindex. Replacefile.extractedContentwith$.documents.getExtractedText(file),stopWords()with$.documents.stopWords(), searches onextractedContentwith$.documents.fulltextSearch(), the fulltext option of the file import with$.documents.processContent(folder), and theapplication.filesystem.indexing.*settings withcontentextraction.*. The Migration chapter of the documentation has the full list. - The Debian package no longer installs Java. It no longer depends on
openjdk-11-jdk-headless, which was outdated: Structr 7 requires GraalVM 25, which is not available as a Debian package. Install GraalVM 25 before installing or upgrading the package, as described in Getting Started.logrotateis now recommended instead of required, like Tesseract for OCR, soapt-get install --no-install-recommendsskips both. - The session cookie is HttpOnly by default.
httpservice.cookies.httponlynow defaults totrue, so JavaScript in the page can no longer readJSESSIONID. The Admin UI and Structr's frontend scripts do not need it. Custom frontend code that reads the cookie, for example to send the session id over its own websocket connection, stops working; sethttpservice.cookies.httponly = falseto keep the old behaviour. Installations that set the value explicitly instructr.confare not affected. - Template expressions with regular expression literals are evaluated now. A
${{ ... }}block containing a regular expression literal with a quote or brace in it, such as/href="([^"]+)'/, used to be skipped silently, so that part of the page rendered empty. The scanner now recognises regular expression literals, so such blocks are evaluated. Check templates whose output changes after the upgrade: they contain a block that was not executed before. A block that is still never closed, for example because of an unbalanced quote, is now logged as a warning naming the template. - Cron steps in the day and month fields count from 1. A step such as
*/3used to be counted wrongly in the day-of-month and month fields:*/3in the months ran in February, May, August and November, and in the days on the 2nd, 5th, 8th and so on. It now counts from the first allowed value, as in the Unix crontab: January, April, July and October, and the 1st, 4th, 7th. Seconds, minutes, hours and days of the week are unaffected. Check cron expressions with a step in the fourth or fifth field. List values outside a field's range, such as0,60in the seconds, are now rejected instead of never matching, and the configuration interface checks expressions with the same rules as the server. - The superuser login is blocked after 3 failed attempts. While the superuser has no second factor, 3 wrong superuser passwords within 15 minutes, counted across all addresses and across the configuration interface,
X-User/X-Passwordand HTTP Basic authentication together, block the superuser everywhere for 15 minutes, including for the right password. Scripts that authenticate as the superuser with a wrong or outdated password trigger it as well.configservlet.login.maxattemptsandconfigservlet.login.lockout.secondsadjust it; 0 seconds turns it off. Withsuperuser.twofactor.enabledon, superuser requests over HTTP need theX-TwoFactor-Codeheader, so scripts that use the superuser stop working; use an admin user for them. - assert() aborts on anything but true. Until 7.0, a
nullargument or a wrong number of arguments madeassert()log an error and let the request continue, even when the condition wasfalse. So$.assert($.me, 401, ...)admitted anonymous users, and$.assert(false, status, msg)withstatusormsgbeingnulllet everyone through. Now only the booleantruepasses; everything else aborts, with the given status code, or 500 when it is missing or not a number. A condition that is not a boolean is logged as a warning. Check scripts that pass objects toassert()instead of comparisons, such as$.assert($.me, ...): write$.assert($.me != null, ...). - Mailboxes are no longer fetched automatically. Until 7.0, the MailService checked every
Mailboxeverymail.updateintervalmilliseconds. It now fetches a mailbox only whenfetchMails()is called on it. An instance that relied on automatic fetching silently stops receiving mail after the upgrade, so add a scheduled task before or right after upgrading: a user-defined function that callsfetchMails()on the mailboxes you need, registered with the CronService, for example
CronService.tasks = fetchAllMailboxes fetchAllMailboxes.cronExpression = 0 */5 * * * *
The Email article ("Fetching on a Schedule") has the complete example. mail.updateinterval has been removed; if it is still set in structr.conf, Structr names its replacement at startup. Three settings are new: mail.connecttimeout (default 30 s), mail.readtimeout (default 60 s) and mail.maxconcurrentfetches (default 4).
- IMAP mailboxes are fetched incrementally. A fetch now reads the messages that arrived since the previous one, oldest first, in batches of
mail.maxemails, so a burst larger than the batch is fetched over several runs instead of being lost. The first fetch after the upgrade takes the newestmail.maxemailsmessages, as before. POP3 mailboxes still read the newestmail.maxemailsevery time. - Duplicates are detected per mailbox. A mail delivered to two of your mailboxes is now stored once for each of them; until 7.0 only the first mailbox to fetch it received a copy.
- New read-only properties on Mailbox:
lastFetchStarted,lastFetchSucceeded,lastFetchError,lastFetchCountandfetchState. Code that counts or lists all properties ofMailboxsees five more. - HTML, SVG and XML files are displayed again instead of downloaded. 7.0 delivered every such file from the file area as a download. 7.1 delivers files as their content type says (RFC 6266) and protects against the case that made the download necessary, an uploaded page running in your application's origin with the session of whoever opens it:
- A file is trusted when an admin uploaded it through the browser; the new read-only
trustedproperty records that. A trusted file is delivered unchanged. - Every other file is untrusted: it is displayed sandboxed, without scripts, forms or framing, and is not cached.
htmlservlet.filedeliverydecides further:auto(default) downloads untrusted files on an instance whoseapplication.instance.stageisproductionand sandboxes them everywhere else;sandbox,attachmentandinlineset one behaviour for all stages. Except withinline, an admin opening an untrusted file from outside the application, or with no referrer, always gets the download. - Files that existed before the upgrade are untrusted. An HTML file that needs its scripts, for example a single-page application kept in the file area, stops working until an admin marks it trusted on the General tab of the file.
trustedandisTemplatecan only be set by an admin now; for anyone else the change is refused with 403, also from scripts that callunlockReadonlyPropertiesOnce().- A download can still be requested explicitly with the
_filenamerequest parameter. - S/MIME mail files are no longer indexed. To fix the mail fetch failures on S/MIME-signed messages, BouncyCastle's S/MIME support for Jakarta Mail (
bcjmail) is no longer shipped. As a side effect, the full-text extraction no longer reads the content of S/MIME-signed or -encrypted mail files stored in the file area; all other file types are unaffected. - Undeclared method parameters are rejected. A method with declared parameters no longer ignores parameters it does not declare: a REST or script call with such a parameter fails with
422 Unprocessable Entityand the method is not executed. Event action mappings still drop them and log a warning. Declare the parameters, or enable "Allow unknown parameters" in the API tab of the method. Methods without declared parameters accept any parameters as before. - Two sort calls behave differently. An order of
'desc'after a name, as in$.find('User', $.predicate.sort('createdDate', 'desc')), now sorts in descending order; before, every value excepttruesorted ascending. An order without a name, as insort(list, true), now sorts by name in descending order. A value after a name that is neither an order nor a known key, like'descending', raises an error instead of being ignored. - Virtual types and the api-builder module are removed. Applications that use virtual types or the api-builder module must replace them before upgrading.
- A fresh installation needs the setup token. Read it from the server log (a line beginning with
Initial setup:), or preset it withsetup.tokenin automated installations. Existing installations with a superuser password are not affected. - Only admins can set skipSecurityRelationships. It can now be enabled for non-admin users as well, but the property is read-only for everyone except admins.
- The sync settings of mounted folders moved to the storage configuration. Scripts that read or write the watch, scan interval or target type properties of a folder, or
sync.deleteStale, need to use the entriessync.watchContents,sync.scanInterval,sync.fileType,sync.folderTypeandsync.onVanishedof the mount's storage configuration.
Dependency Updates
- logback, kafka-clients, BouncyCastle, mina-core, commons-configuration2, jackson, jsoup and netty are updated to their latest patch versions to fix known vulnerabilities. All BouncyCastle jars are kept on one version, and
bcjmailis no longer shipped. - Monaco editor 0.55.1; terser 5.51.2 and csso 5.0.5 are bundled for
minify().