| CARVIEW |
Select Language
HTTP/2 200
date: Sat, 26 Jul 2025 20:31:02 GMT
content-type: text/html; charset=utf-8
cache-control: no-cache
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com release-assets.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com copilotprodattachments.blob.core.windows.net/github-production-copilot-attachments/ github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.githubassets.com github.com/assets-cdn/worker/ github.com/assets/ gist.github.com/assets-cdn/worker/
referrer-policy: no-referrer-when-downgrade
server-timing: pull_request_layout-fragment;desc="pull_request_layout fragment";dur=512.124748,conversation_content-fragment;desc="conversation_content fragment";dur=1955.353896,conversation_sidebar-fragment;desc="conversation_sidebar fragment";dur=436.147657,nginx;desc="NGINX";dur=1.011051,glb;desc="GLB";dur=101.156988
strict-transport-security: max-age=31536000; includeSubdomains; preload
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, X-Requested-With,Accept-Encoding, Accept, X-Requested-With
x-content-type-options: nosniff
x-frame-options: deny
x-voltron-version: a2eb102
x-xss-protection: 0
server: github.com
content-encoding: gzip
accept-ranges: bytes
set-cookie: _gh_sess=ZRIZnC4zWASlRdmrei8MLANJseFwyMdvAwjvXCz0qw1JjFaAZF5mnhOzNULEwWKpM1F6psHELMMVoB%2BEFgaigQf3OPsEqThPnSIyg5v0NLEaH8TkJvVIvfTSSILxko9bonY0JaagfZ2irCRmmhKpml%2BZo%2F4Xq65QjjAv%2F7fJersiuS3QMV%2FiN8rzeRYx5JhHqTKUYvqF7srop3TlEGJGCyaMEcSb5Tc9iVCNqGt3vpaL9zUicMmqPRObb6DEYz8VPSOGjktKKVkvGg1ipFaAWw%3D%3D--S7VG6s5r%2B0UMnrq7--tFsB7BW7Ql6SfJEMPbjs2w%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.354813533.1753561860; Path=/; Domain=github.com; Expires=Sun, 26 Jul 2026 20:31:00 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Sun, 26 Jul 2026 20:31:00 GMT; HttpOnly; Secure; SameSite=Lax
x-github-request-id: 96A0:1A1F38:8E0CEF:B775E8:68853B04
DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs by aparajit-pratap · Pull Request #16273 · DynamoDS/Dynamo · GitHub
aparajit-pratap
changed the title
Continued from PR 16255: Improve Library Loading of many DYFs
DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs
Jun 6, 2025
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4)
aparajit-pratap
changed the title
DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs
DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs [WIP]
Jun 6, 2025
Skip to content
Navigation Menu
{{ message }}
-
Notifications
You must be signed in to change notification settings - Fork 658
DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs #16273
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
aparajit-pratap
merged 36 commits into
DynamoDS:master
from
aparajit-pratap:customnode-loading-perf
Jul 10, 2025
Merged
DYN-9011: Contd. from PR 16255 - Improve Library Loading of many DYFs #16273
aparajit-pratap
merged 36 commits into
DynamoDS:master
from
aparajit-pratap:customnode-loading-perf
Jul 10, 2025
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mance-library-json-loading
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
aparajit-pratap
changed the title
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See the ticket for this pull request: https://jira.autodesk.com/browse/DYN-9011
aparajit-pratap
changed the title
aparajit-pratap
commented
Jun 10, 2025
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
aparajit-pratap
commented
Jun 27, 2025
|
In terms of the pruning of cache items that represent files that are no longer on disk or no longer match the hash maybe something like this: public void Prune()
{
foreach (var key in cache.Keys.ToList())
{
//get the path from the key
int idx = key.IndexOf(".dyf", StringComparison.OrdinalIgnoreCase);
string path = idx >= 0 ? key.Substring(0, idx + 4) : string.Empty;
if (File.Exists(path))
{
//Check the files write time and length (same mechanism as original key)
var fileinfo = new FileInfo(path);
var lastWriteTime = fileinfo.LastWriteTimeUtc;
var length = fileinfo.Length;
var newKey = path + lastWriteTime.ToString() + length.ToString();
if(newKey != key)
{
//If keys do not match then the file has been updated, remove from cache
cache.Remove(key);
}
}
else
{
//file no longer exists, remove from cache
cache.Remove(key);
}
}
}Not sure exactly when to tigger. We could do it on exit but that might take a while. Maybe a background task? Again not sure when to tigger but maybe only needs to be done once per user session |
…Dynamo into customnode-loading-perf
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
reddyashish
reviewed
Jul 7, 2025
|
This passes here except for one test (which I think is unrelated and failing due to it being flaky) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR enhances library loading performance by caching custom node metadata, adds a fallback error message resource, and refactors related parsing logic.
- Introduced
JsonCache<T>to store and serializeCustomNodeInfoon disk with stale‐entry cleanup. - Updated
TryGetInfoFromPathto use the cache and unified JSON/XML parsing paths. - Added a new resource string for unknown file‐processing errors and extended
CustomNodeDefinitionto support JSON‐based parsing.
Reviewed Changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/DynamoCore/Properties/Resources.resx | Added UnknownErrorProcessingFile resource for fallback error messaging. |
| src/DynamoCore/Properties/Resources.en-US.resx | Added US variant for the unknown error resource. |
| src/DynamoCore/Models/DynamoModel.cs | Removed existing‐element sync logic in InitializeCustomNodeManager and simplified early return. |
| src/DynamoCore/Core/CustomNodeManager.cs | Added JsonCache<T> class, initialized cache on startup, and offloaded stale‐entry removal. |
| src/DynamoCore/Core/CustomNodeDefinition.cs | Added manual JSON parsing (GetFromJsonDocument) and [JsonConstructor] for CustomNodeInfo. |
Files not reviewed (1)
- src/DynamoCore/Properties/Resources.Designer.cs: Language not supported
Comments suppressed due to low confidence (4)
src/DynamoCore/Core/CustomNodeManager.cs:44
- New caching logic in
JsonCache<T>and its interaction with disk I/O and stale‐entry removal require unit tests to verify correct serialization, deserialization, and cleanup behavior.
private class JsonCache<T>
src/DynamoCore/Models/DynamoModel.cs:1517
- The removal of the block that synced existing
CustomNodeSearchElementinstances (and updated them) means updates to already‐loaded nodes are no longer applied. Consider restoring that logic to keep the search model in sync when a node is reloaded.
return;
src/DynamoCore/Core/CustomNodeManager.cs:861
- Variable
exis declared inside nested scopes and is out of scope here, causing a compile error. DeclareException ex;before thetryblock or consolidate error handling to ensureexis available at this point.
if (ex == null)
src/DynamoCore/Core/CustomNodeDefinition.cs:346
- [nitpick] Changing
IsVisibleInDynamoLibraryfrom a read‐only property to a public setter may break invariants. If external mutation is unintended, revert to a private setter or validate assignments.
public bool IsVisibleInDynamoLibrary { get; set; }
reddyashish
reviewed
Jul 9, 2025
|
hey @aparajit-pratap - for the dynamo as a service case - I don't think caching will help us (at least not with our single use architecture) and writing to disk will likely be a waste of time for us. Two questions with that context:
|
@mjkkirschner, thanks for the review. I'm trying to measure the overall performance gain (if any) with this PR; will share the results shortly. I haven't characterized the overhead related to saving the cache, no. No, there's currently no startup config to control if the cache is generated but I could add one. I can see why for DAAS it might not be that useful, as with the use-and-throw containers, you won't be reloading the same packages and dyf's repeatedly. |
reddyashish
reviewed
Jul 9, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changes look good.
zeusongit
approved these changes
Jul 9, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
You can’t perform that action at this time.
Purpose
Continued from PR #16255 : Improve Library Loading of many DYFs.
Some metrics (3 tries):
Customnode loading performance (clockwork package = ~453 custom nodes):
Deserialize cache-file when empty - ~0 ms
Deserialize cached file - 117/39/48 ms
Construction of nodes from cache - 175/105/89 ms
Construction from JSON - 1000/368/376 ms
Serialize cache to file - 31/20/1 ms
It can be seen that in general using the cache (deserialization of cached file + construction of nodes from in-memory cache) takes less than half the time it would take to construct the nodes from JSON deserialization (and that is after the optimizations in this PR for JSON deserialization).
Declarations
Check these if you believe they are true
*.resxfilesRelease Notes
(FILL ME IN) Brief description of the fix / enhancement. Use N/A to indicate that the changes in this pull request do not apply to Release Notes. Mandatory section
Reviewers
(FILL ME IN) Reviewer 1 (If possible, assign the Reviewer for the PR)
(FILL ME IN, optional) Any additional notes to reviewers or testers.
FYIs
(FILL ME IN, Optional) Names of anyone else you wish to be notified of