Remote projects with GIt LFS

JohnMoshakis · November 29, 2020, 3:32am

Hi,

Is this supported ?

I went to where ebuild has the repo and did git lfs install and it says

Updated git hooks.
Git LFS initialized.

When I build I get

D: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld -arch arm64 -o “/Users/JohnMoshakis/Library/Application Support/RemObjects Software/EBuild/Obj/RealmiOSApplication-93A1D62F6E1CCA3EB4E730C04ECF7FEE2B46B664/Debug/Toffee-iOS/Device/arm64/RealmiOSApplication” “/Users/JohnMoshakis/Library/Application Support/RemObjects Software/EBuild/Obj/RealmiOSApplication-93A1D62F6E1CCA3EB4E730C04ECF7FEE2B46B664/Debug/Toffee-iOS/Device/arm64/RealmiOSApplication.a” “/Users/JohnMoshakis/Library/Application Support/RemObjects Software/EBuild/Obj/RealmiOSApplication-93A1D62F6E1CCA3EB4E730C04ECF7FEE2B46B664/Debug/Toffee-iOS/Device/arm64/RealmiOSApplication.o” -lSystem -lobjc -lToffee -lxml2 -L/Users/JohnMoshakis/Documents/Elements/References/Toffee/iOS -framework CloudKit -framework CoreFoundation -framework CoreGraphics -framework Foundation -framework Intents -framework Realm -framework UIKit -framework UserNotifications “-F/Users/JohnMoshakis/Library/Application Support/RemObjects Software/EBuild/Obj/Realm-9419A8B1244B42BDC9CB71A15D58B33F4F8380BA/Unknown/Toffee-iOS/Device” -ObjC -dynamic -demangle -force_load /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.ios.a -dead_strip -rpath @executable_path -rpath @executable_path/Frameworks -syslibroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.2.sdk -F/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/Library/Frameworks
E: ld: warning: ignoring file /Users/JohnMoshakis/Library/Application Support/RemObjects Software/EBuild/Obj/Realm-9419A8B1244B42BDC9CB71A15D58B33F4F8380BA/Unknown/Toffee-iOS/Device/Realm.framework/Realm, building for iOS-arm64 but attempting to link with file built for unknown-unsupported file format ( 0x76 0x65 0x72 0x73 0x69 0x6F 0x6E 0x20 0x68 0x74 0x74 0x70 0x73 0x3A 0x2F 0x2F )
> ld: Undefined symbols for architecture arm64:
> ld: “OBJC_CLASS$_RLMRealm”, referenced from:
> ld: objc-class-ref in RealmiOSApplication.a(MainViewController-27a04cc4d23404329c3e844021c9c123.o)
> ld: “OBJC_CLASS$_RLMRealmConfiguration”, referenced from:
> ld: objc-class-ref in RealmiOSApplication.a(AppDelegate-d556fda9a4046af5a037b68ec31609be.o)
> ld: “OBJC_CLASS$_RLMObject”, referenced from:
> ld: OBJC_CLASS$_DataObject in RealmiOSApplication.a(DataObject-c280a8dbcdc876442e3ba2759dcbd565.o)
> ld: “OBJC_METACLASS$_RLMObject”, referenced from:
> ld: OBJC_METACLASS$_DataObject in RealmiOSApplication.a(DataObject-c280a8dbcdc876442e3ba2759dcbd565.o)
E: ld: symbol(s) not found for architecture arm64

Realm is the file that I used with lfs.

Cheers,
John

mh · November 29, 2020, 3:18pm

Sounds like this is not a valid binary. what does file say?

JohnMoshakis · December 6, 2020, 6:09am

The file starts off as a pointer

I was able to get it working by going into the ebuild git repo and doing “git lfs install”

Reading that link seems to imply once it’s turned on it should work using regular commands.

It didn’t seem to so I did a “git lfs pull” and then the file changed from the reference to the actual one.

I don’t suppose it would be possible to support this from within Fire ?

I have a git repo with all my import projects and the associated framework files. Im using remote references for this. It seemed to me like a great idea to share import projects between ios apps but I didnt realize that git has a 100mb file limit and one of the files in the latest realm framework is greater than this.

Cheers,
John

mh · December 6, 2020, 12:46pm

Sure. I’m not very familiar with LFS support int git. is it safe to just always do a “git lfs pull” (even if its not installed), or else, is there a clean way to find if its necessary?

JohnMoshakis · December 7, 2020, 3:55am

You need to install it first. There is a lfs folder under the .git folder, I was wondering if that could be used to determine if its installed ?

Im still not entirely sure if you need the extra lfs commands or not. I haven’t been able to get it working without doing an extra git lfs pull.

mh · December 8, 2020, 12:13pm

I’ll have a look.

do you have a public repo with a remote project that uses LFS that I could use to test against?

—marc

JohnMoshakis · December 8, 2020, 1:47pm

github.com

mosh/ImportSolution/blob/master/Realm/iOS Simulator/Realm.framework/Realm

version https://git-lfs.github.com/spec/v1
oid sha256:a45609d916076c16687afe4214914020f710e08525ea788f5900b9dd461e2b4a
size 119156892

and

github.com

mosh/ImportSolution/blob/master/Realm/iOS/Realm.framework/Realm

version https://git-lfs.github.com/spec/v1
oid sha256:a45609d916076c16687afe4214914020f710e08525ea788f5900b9dd461e2b4a
size 119156892

are the large files.

mh · December 8, 2020, 2:16pm

If I add this remote project ref to an iOS project, should it fail to link, out of the box? it doesn’t, here…

that said, the file isn not the proper binary:

➜  Cherry:~/Library/Application Support/RemObjects Software/EBuild/Packages/EBuild/github.com.mosh.ImportSolution/Realm/iOS/Realm.framework git:(master) # file Realm
Realm: ASCII text
➜  Cherry:~/Library/Application Support/RemObjects Software/EBuild/Packages/EBuild/github.com.mosh.ImportSolution/Realm/iOS/Realm.framework git:(master) # cat Realm 
version https://git-lfs.github.com/spec/v1
oid sha256:a45609d916076c16687afe4214914020f710e08525ea788f5900b9dd461e2b4a
size 119156892
➜  Cherry:~/Library/Application Support/RemObjects Software/EBuild/Packages/EBuild/github.com.mosh.ImportSolution/Realm/iOS/Realm.framework git:(master) #

however, git ofd install does not seem to be a valid command; I assume I’d need to install this https://git-lfs.github.com first? I’m hesitant to do this right now, as I understand it could potentially affect all git repros (also, it uses brew, which isn’t supported on Apple Silicon yet)…

What I can do is implement the following changes “blind”, assuming you can confirm the exact commands/folders:

after checkout, if ./git/lfs exists as a folder on disk, I will run git lfs pull after each regular pull.

With the, I assume an initial build (ie a clean checkout) would still fail the first time, until there user activates LFS for the repo manually (no way to detect that automatically, really?), but once it’s done, subsequent builds should pull correctly.

how does that sound?

JohnMoshakis · December 8, 2020, 8:17pm

Its git lfs install and yes you need to install lfs from https://git-lfs.github.com

I think when you initially clone you would need to call git lfs install, and then each time you pull do a git lfs pull.

mh · December 10, 2020, 11:57am

But how would I know when to do that?

JohnMoshakis · December 10, 2020, 12:24pm

Yes thats true. Would it be possible to add an attribute ?

<RemoteProjectReference Include="github.com/Mosh/iOSApp.Core/libiOSApp.Core.elements:*">
  <Private>True</Private>
  <lfs>True</lfs>
</RemoteProjectReference>

Then you also wouldnt need to check for the lfs folder

mh · December 10, 2020, 12:35pm

I can do that. SO if thats present, I do lsf install on clone and lfs pull on pull. Will add for tomorrow (blind), let me know how it goes,.

mh · December 10, 2020, 12:47pm

Done. the setting will be <LFS>True</LFS>, upper-case.

JohnMoshakis · December 10, 2020, 1:11pm

I just need to download an upcoming zip file ?

mh · December 10, 2020, 1:11pm

It’s merged now, so the next build should have this, yes.

JohnMoshakis · December 11, 2020, 3:36am

If I delete the repo and build I get this

git: Updating files: 100% (2795/2795), done.
git: Already on ‘master’
git: Your branch is up to date with ‘origin/master’.
git: WARNING: error running /Applications/Xcode.app/Contents/Developer/usr/libexec/git-core/git ‘config’ ‘–global’ ‘–replace-all’ ‘filter.lfs.clean’ ‘git-lfs clean – %f’: ‘fatal: $HOME not set’ ‘exit status 128’
git: Run git lfs install --force to reset git config.
E: There was a problem obtaining remote project ‘github.com/mosh/ImportSolution/Realm.elements:*’ referenced from ‘RealmiOSApplication’.

mh · December 11, 2020, 12:10pm

Ok, what does that mean for us, should I always use --force, jic?

JohnMoshakis · December 11, 2020, 3:49pm

I don’t think so.

Previously I deleted the folder and did a build. I then went into the folder and did a git lfs install, followed by a git lfs pull. After that a build would be successful.

Are you running any lfs commands before the lfs install ?

mh · December 11, 2020, 4:30pm

No.

        if lFolder.FolderExists then begin
          if Path.Combine(lFolder, ".git").FolderExists then begin
            aContext.Logger.LogDetail($"Pulling {aUrl} to {lFolder}");
            if not RunGit(aContext, ["checkout", aBranch], lFolder) then exit false;
            if not RunGit(aContext, ["pull"], lFolder) then begin
              aContext.Logger.Warning("Pull for git repository {0} failed.", aUrl);
              if aForcePull then
                exit false;
            end;
            if aLFS then
              if not RunGit(aContext, ["lfs", "pull"], lFolder) then exit false;
          end
          else begin
            aContext.Logger.Error($"Folder '{lFolder}' already exists, but is not a git repository.");
            exit false;
          end;
        end
        else begin
          aContext.Logger.LogDetail($"Cloning {aUrl} to {lFolder}");
          if not RunGit(aContext, ["clone", aUrl, lFolder]) then exit false;
          if not RunGit(aContext, ["checkout", aBranch], lFolder) then exit false;
          if aLFS then begin
            if not RunGit(aContext, ["lfs", "install", "--force"], lFolder) then exit false; // --force added now
            if not RunGit(aContext, ["lfs", "pull"], lFolder) then exit false;
          end;
        end;

JohnMoshakis · December 12, 2020, 1:14am

Where is filter.lfs.clean’ ‘git-lfs clean coming from ?