Improve use of asset validation errors

[jjnesbitt]

• Closes problem rendering many asset validation errors #2540
• Closes Remove use of Asset VALIDATING status if not used #1653
• Closes Dandiset info endpoint large response sizes #1214

The primary purpose of this PR is to address an issue with asset validation errors for zarr assets. The existing behavior is to share the same status between assets that are in the process of validation, and zarr assets that cannot be validated yet (due to the underlying zarr not being finalized).

This is observed mostly on dandiset 000108, however other dandisets have experienced this as well. This change won't actually address the underlying problem, as the solution to that is simply to finalize the zarrs backing those assets (something I will address separately). However, it will fix the asset validation error that is shown, to more accurately describe the problem.

There are several secondary improvements:

• Remove the deprecated VALIDATING status on the Asset model
• Fix the broken asset validation UI
• Separate out asset validation errors into their own endpoint, removing them from the version info endpoint.
• Enabled by the above improvement, remove the truncation of asset validation errors (originally introduced in Truncate asset validation errors/fix memory usage #2548)

[jjnesbitt]

I still need to address CLI integration...

[yarikoptic]

should not be merged with this AFAIK... I wish there was a comment about PR which to check if this branch was merged/released

Suggested change

	uvx --with "dandi[test] @ git+https://github.com/dandi/dandi-cli@use-new-asset-validation-errors"
	uvx --with "dandi[test] @ git+https://github.com/dandi/dandi-cli"

[yarikoptic]

Found it (draft ATM):

• Use new asset_validation_errors endpoint dandi-cli#1823

[yarikoptic]

remove testing against some dev branch of dandi-cli

[jjnesbitt]

This PR is still blocked on CLI integration. I do not have time to address this in the short term, so I will put it back into draft.

[yarikoptic]

I do not see 'restart' button somehow but should be ok if redone now (dandi-cli vise)

[yarikoptic]

note: I tuned description to put "Closes #XXX" into itemized list so we could see the titles of those issues to be closed

However, it will fix the asset validation error that is shown, to more accurately describe the problem.

you left me hanging there ;) could you visualize how it looked before in UI and now?

Screenshot for added new endpoint in swagger?

I am yet to grasp why we need a new endpoint (and thus changes to dandi-cli) as we already have per asset one:

and not some option for an existing endpoint to filter if so desired for UI -- was such an option considered? (we already have many good options for other endpoints, as eg. filtering in /dandisets/ etc)

[jjnesbitt]

I am yet to grasp why we need a new endpoint (and thus changes to dandi-cli) as we already have per asset one:

The per asset validation error endpoint is not helpful if you want to view all asset validation errors.

and not some option for an existing endpoint to filter if so desired for UI -- was such an option considered? (we already have many good options for other endpoints, as eg. filtering in /dandisets/ etc)

The dandisets endpoint you mentioned is perhaps the most clunky and inefficient endpoint in the entire API (and to which we've had numerous past bugs), precisely because it attempts to do too many things at once via query params and filtering.

Right now, the asset validation errors are bundled into the version info endpoint, which is used in several ways, and is overall an important endpoint. Since there may be many asset validation errors for a given version, if all errors are returned, this can can cause timeouts or otherwise slow response times. To avoid this, we artificially limit the number of asset validation errors that are returned from the version info endpoint. However, this causes issues, namely an incomplete and misleading picture of what asset validation errors are present on a version.

Separating out the asset validation errors into their own endpoint removes this burden from the main version info endpoint, and allows for the full list of asset validation errors to be returned from the new asset validation errors endpoint, without affecting the return of other data.