I believe that the RDF to object conversion, step 2.4.3, has a blind spot.
This is in the case useNativeTypes is used:

if the datatype IRI of value equals xsd:integer or xsd:double and its lexical form is a valid xsd:integer or xsd:double according [XMLSCHEMA11-2], set converted value to the result of converting the lexical form to a JSON number.

The problem is: some valid literals of these datatypes may fail to convert to JSON numbers, e.g. "+INF"^^xsd:double, or very large integers.

The spec is currently silent about that. We should either specify that an error must be raised, or that useNativeType should be ignored in that case, and a regular value object should be generated. I personally prefer the 2nd option.