Skip to content

fwrite: avoid converting matrix input to data.table #3188

Open

Description

@mattdowle

PR #3125 implemented #2613 to enable to fwrite to accept matrix. This was merged in v1.12.0. However it converts matrix to data.table which can be costly.
The convert can be avoided as follows.
fwriteR.c contains this :

for (int j=0; j<args.ncol; j++) {
    SEXP column = VECTOR_ELT(DFcoerced, j);
    if (args.nrow != length(column))
      error("Column %d's length (%d) is not the same as column 1's length (%d)", j+1, length(column), args.nrow);
    int32_t wf = whichWriter(column);
    if (wf<0) {
      error("Column %d's type is '%s' - not yet implemented in fwrite.", j+1, type2char(TYPEOF(column)));
    }
    args.columns[j] = (wf==WF_CategString ? column : (void *)DATAPTR(column));
    args.whichFun[j] = (uint8_t)wf;
    if (TYPEOF(column)==VECSXP && firstListColumn==0) firstListColumn = j+1;
  }

This loop populates the pointers in arg.columns[] which are then passed to fwrite.c. That DATAPTR returns a pointer to where the data (e.g. int *, double *) for the R vector starts. In the case of matrix, this args.columns[] just needs to be populated with offsets into matrix vector. In R a matrix is a single very long vector just with dimension attribute attached. A matrix in R is columnar, just like a data.table, so there doesn't need to be a transpose. It should be fairly simple to do and work well.

Careful to add tests for a factor matrix as well as a character matrix.

I described how to do it in case @fparages or @MichaelChirico wanted to give it a go.

Activity

  1. added this to the 1.12.0 milestone on Dec 6, 2018
  2. MichaelChirico commented on Dec 6, 2018

    @MichaelChirico
    Member

    Agree the as.data.table approach is a bit of a have and this is right way to do things.

    How does one point to matrix column names in C API?

    Don't think I'll have time before next year to work on something that for me & my lack of C experience will end up being quite time consuming (only mentioning because I see the 1.12.0 milestone and know you mentioned maybe pushing out soon given the memory fixes)

  3. mattdowle commented on Dec 6, 2018

    @mattdowle
    MemberAuthor

    Sure thing.

    args.columns[j] = (char *)DATAPTR(matrix) + j*size*nrow;
    

    where size is 4 for int and 8 for double, for example.

  4. jangorecki commented on Dec 6, 2018

    @jangorecki
    Member

    instead of DATAPTR isn't it more appropriate to use switch(TYPEOF(matrix)) ... INTEGER(matrix) ... REAL(matrix)?

  5. mattdowle commented on Dec 6, 2018

    @mattdowle
    MemberAuthor

    It's more convenient and appropriate to use DATAPTR. But we're not supposed to is more the point because it's not part of the R API. I wish it were and I don't see why it isn't. It should be the switch() way to be compliant, yes.
    We could create our own dataptr() function that does the switch. Seems silly to have to do so though when DATAPTR could just be added to the R API. There must be a good reason why it isnt.
    Maybe now that INTEGER, REAL and DATAPTR are no longer ever macros, even under USE_RINTERNALS, the original reason for not exposing DATAPTR in R API might have gone away.

  6. modified the milestones: 1.12.0, 1.12.2 on Jan 6, 2019
  7. raneameya commented on Jan 23, 2019

    @raneameya

    I'd like to draw attention to the fact that matrices can have rownames which in the current version are stripped away in the as.data.table conversion. May I please ask that rownames be written as the first column by default if it is sensible? If not, maybe an additional argument to include rownames, like as.data.table?

  8. mattdowle commented on Jan 23, 2019

    @mattdowle
    MemberAuthor

    Re rownames: sure thing. If rownames are the default 1:nrow then can we assume please they aren't really rownames and not write them by default? Internally R uses c(NA,INT_MIN) to represent default row names efficiently, so it's safe and efficient to test that attribute to see if row.names is a length 2 integer vector holding those values. With an argument to explicitly specify row names too of course; we're just talking about the default here.

  9. raneameya commented on Jan 23, 2019

    @raneameya

    Thanks Matt!

  10. modified the milestones: 1.12.2, 1.12.4 on Jan 24, 2019
  11. modified the milestones: 1.12.4, 1.13.0 on Jul 25, 2019
  12. modified the milestones: 1.12.7, 1.12.9 on Dec 8, 2019
  13. modified the milestones: 1.13.1, 1.13.3 on Oct 17, 2020
  14. modified the milestones: 1.14.3, on Jul 19, 2022
  15. modified the milestones: , 1.15.1 on Oct 29, 2023
  16. modified the milestones: 1.16.0, 1.17.0 on Jul 10, 2024
  17. modified the milestones: 1.17.0, 1.18.0 on Dec 9, 2024
  18. modified the milestones: 1.18.0, 1.19.0 on Nov 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions