fix: add custom robots.txt (#4823)
## Changes Made
add custom robots.txt for google search indexing
## Related Issues
n/a
## Checklist
- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)

feat: Spread actor udf tasks (#4802)
## Changes Made
Spread actor UDF tasks and pass all the handles to the swordfish worker,
let it choose which actor to use based on the location.
## Related Issues
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
## Checklist
- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-159.us-west-2.compute.internal>

fix: Ignore partition refs if they are empty (#4766)
## Changes Made
This PR prevents propagation of empty partition refs for subsequent
tasks
## Related Issues
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
## Checklist
- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)

fix(docs): Updated docs to correct calculate max rows per partition (#…
…4744)
## Changes Made
The monotonically increasing ID document has a small error in maximum
rows per partition calculation. It should say `2^36` instead of `2^40`.

chore: keep same ruff version with pre-commit-config (#4673)

feat(flotilla): Enable flotilla by default (#4641)

chore: Use scarf only for analytics (#4595)
## Changes Made
<!-- Describe what changes were made and why. Include implementation
details if necessary. -->
## Related Issues
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
## Checklist
- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)

feat: add new "dashboard" metrics subscriber (#4527)
## Changes Made
- adds a new "dashboard" metrics subscriber that'll make a rest call to
`DAFT_DASHBOARD_METRICS_URL` with the provided metrics information
- small internal refactoring to split `runtime_stats` into module and
submodules
## Related Issues
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
## Checklist
- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)

feat(flotilla): Use max sources config to determine files per partiti…
…on (#4535)
## Changes Made
Allow the number of files per partition in swordfish to be configurable
by config. For now.
Ideally the scheduler or optimizer will be able to size them better
## Related Issues
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
## Checklist
- [ ] Documented in API Docs (if applicable)
- [ ] Documented in User Guide (if applicable)
- [ ] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [ ] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)

feat: url download for unity catalog volumes (#4476)
## Changes Made
Enable `Expression.url.download` to download from Unity Catalog volumes.
## Related Issues
none
## Checklist
- [x] Documented in API Docs (if applicable)
- [x] Documented in User Guide (if applicable)
- [x] If adding a new documentation page, doc is added to
`docs/mkdocs.yml` navigation
- [x] Documentation builds and is formatted properly (tag @/ccmao1130
for docs review)