As an alternative to SQL Server (which would probably be sensible). You could move the files to OneDrive for Business (attached to a Power BI Workgroup), you wouldn't need the Gateway then and can enumerate all the files on one go.
These files could be synced from on premise using ODFB automatically.
HTH
Sacha