Efficient way to map a Google drive content with fewer java API calls

49 views Asked by At

Hello everyone i have a code that have for purpose to list files present of a shared drive (to download and create same folders paths later) Currently i do something like this :

  HashMap<String, Strin> foldersPathToID = new HashMap<>();

//searching all folders first saving their IDs 
searchAllFoldersRecursive(folderName.trim(), driveId, foldersPathToID);
  //then listing files in all folders
    HashMap<String, List<File>> pathFile = new HashMap<>();
    for (Entry<String, String> pathFolder : foldersPathToID.entrySet()) {
        List<File> result = search(Type.FILE, pathFolder.getValue());
        if (result.size() > 0) {
            String targetPathFolder = pathFolder.getKey().trim();
            pathFile.putIfAbsent(targetPathFolder, new ArrayList<>());
            for (File file : result) {
                pathFile.get(targetPathFolder).add(file);
            }
        }
    }

Where the recursive method is :

private static void searchAllFoldersRecursive(String nameFold, String id, HashMap<String, String> map) throws IOException, RefreshTokenException {   
 map.putIfAbsent(nameFold, id);
    List<File> result;

    result = search(Type.FOLDER, id);

    // dig deeper
    if (result.size() > 0) {
        for (File folder : result) {
            searchAllFoldersRecursive(nameFold + java.io.File.separator + normalizeName(folder.getName()),
                    folder.getId(), map);
        }
    }
} 

And the search function is :

    private static List<com.google.api.services.drive.model.File> search(Type type, String folderId)
            throws IOException, RefreshTokenException {
        String nextPageToken = "go";
        List<File> driveFolders = new ArrayList<>();
        com.google.api.services.drive.Drive.Files.List request = service.files()
                .list()
                .setQ("'" + folderId
                        + "' in parents and mimeType" + (type == Type.FOLDER ? "=" : "!=")
                        + "'application/vnd.google-apps.folder' and trashed = false")
                .setPageSize(100).setFields("nextPageToken, files(id, name)");

        while (nextPageToken != null && nextPageToken.length() > 0) {
            try {
                FileList result = request.execute();
                driveFolders.addAll(result.getFiles());
                nextPageToken = result.getNextPageToken();
                request.setPageToken(nextPageToken);
                return driveFolders;
            } catch (TokenResponseException tokenError) {
                if (tokenError.getDetails().getError().equalsIgnoreCase("invalid_grant")) {
                    log.err("Token no more valid removing it Please retry");
                    java.io.File cred = new java.io.File("./tokens/StoredCredential");
                    if (cred.exists()) {
                        cred.delete();
                    }
                    throw new RefreshTokenException("Creds invalid will retry re allow for the token");
                }
                log.err("Error while geting response with token for folder id : " + folderId, tokenError);
                nextPageToken = null;
            } catch (Exception e) {
                log.err("Error while reading folder id : " + folderId, e);
                nextPageToken = null;
            }

        }
        return new ArrayList<>();
    }

I'm sure there is a way to do a proper mapping of everyfile (with the folder tree paths) with few api call (maybe even one call ?) because here in my version i do a lot of time the call service.files().list().setQ("'" + folderId+ "' in parents and mimeType" + (type == Type.FOLDER ? "=" : "!=") + "'application/vnd.google-apps.folder' and trashed = false").setPageSize(100).setFields("nextPageToken, files(id, name)");

at least one time per subfolder ... And it's taking a really long time to search everything recursively. In the end the mapping is taking more time than the download itself ...

i searched the documentation and also here but didn't find anything to list all a drive with one library call any ideas ?

I want to get all the files as well as their relative paths in a shared GoogleDrive with the dedicated java API but with the few calls as possible.

Thanks in advance for your time and answers

0

There are 0 answers