Video in repo

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Video in repo

Augustin Trancart
Hi there!

Quick question with the release in sight: a video of 24M has been added (in
aabc913) and removed later (in b321c01) in the repository. This makes the clone
of the repository quite long even though the video is not in master any more.
Should we rewrite the history to get rid of it?

It's possible, but it implies a force-push on master. It means that everybody
will need to use "git rebase --onto master <commit-range>" for all their
branches after this push is done and force-push them (and maybe a git reset
--hard on their local master, I'm not sure how clever git is nowadays).

It's a low risk process (we'll keep a copy of the old master around), but it's a
bit of a hassle and requires synchronization between us. I believe it is
possible, but if we want to do it, we'd better do it now, before success hits us
hard and we have thousands of contributors (yeah, I'm an optimistic kind of guy
;-) )

Thoughts?

--
Augustin Trancart - Oslandia
[hidden email]


_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Pierre-Eric Pelloux-Prayer
+1 let's do this before the first release.


Le 29/06/2017 à 11:32, Augustin Trancart a écrit :

> Hi there!
>
> Quick question with the release in sight: a video of 24M has been added (in
> aabc913) and removed later (in b321c01) in the repository. This makes the clone
> of the repository quite long even though the video is not in master any more.
> Should we rewrite the history to get rid of it?
>
> It's possible, but it implies a force-push on master. It means that everybody
> will need to use "git rebase --onto master <commit-range>" for all their
> branches after this push is done and force-push them (and maybe a git reset
> --hard on their local master, I'm not sure how clever git is nowadays).
>
> It's a low risk process (we'll keep a copy of the old master around), but it's a
> bit of a hassle and requires synchronization between us. I believe it is
> possible, but if we want to do it, we'd better do it now, before success hits us
> hard and we have thousands of contributors (yeah, I'm an optimistic kind of guy
> ;-) )
>
> Thoughts?
>
>
>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>
_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
In reply to this post by Augustin Trancart
I should clarify something: permanently keeping a copy of old master (or any
branch that contains this video) would prevent the problem to be resolved. So at
the end, every one of these branch should either get deleted or rebased for this
to have any effect. Then we'll let git gc do its work. Let me know if you think
that won't work.

Augustin Trancart - Oslandia
[hidden email]

On 29/06/2017 11:32, Augustin Trancart wrote:

> Hi there!
>
> Quick question with the release in sight: a video of 24M has been added (in
> aabc913) and removed later (in b321c01) in the repository. This makes the clone
> of the repository quite long even though the video is not in master any more.
> Should we rewrite the history to get rid of it?
>
> It's possible, but it implies a force-push on master. It means that everybody
> will need to use "git rebase --onto master <commit-range>" for all their
> branches after this push is done and force-push them (and maybe a git reset
> --hard on their local master, I'm not sure how clever git is nowadays).
>
> It's a low risk process (we'll keep a copy of the old master around), but it's a
> bit of a hassle and requires synchronization between us. I believe it is
> possible, but if we want to do it, we'd better do it now, before success hits us
> hard and we have thousands of contributors (yeah, I'm an optimistic kind of guy
> ;-) )
>
> Thoughts?
>
>
>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Thomas Broyer
In reply to this post by Augustin Trancart
15 months ago (or so), big files were added to the project and then removed, and I also floated the idea of a history rewrite. I never got the "go" so it hasn't been done. Iff we go that road for the video file, I suggest we also get rid of those old commits.

I don't have a strong opinion whether this should be done, but it's clear that it's now or never.

I'd suggest we eventually find a way to somehow reject pull requests (we apparently can't reject commits/pushes) with big files (or many smaller files!); possibly an additional check in Travis, I don't know.

2017-06-29 11:32 GMT+02:00 Augustin Trancart <[hidden email]>:
Hi there!

Quick question with the release in sight: a video of 24M has been added (in
aabc913) and removed later (in b321c01) in the repository. This makes the clone
of the repository quite long even though the video is not in master any more.
Should we rewrite the history to get rid of it?

It's possible, but it implies a force-push on master. It means that everybody
will need to use "git rebase --onto master <commit-range>" for all their
branches after this push is done and force-push them (and maybe a git reset
--hard on their local master, I'm not sure how clever git is nowadays).

It's a low risk process (we'll keep a copy of the old master around), but it's a
bit of a hassle and requires synchronization between us. I believe it is
possible, but if we want to do it, we'd better do it now, before success hits us
hard and we have thousands of contributors (yeah, I'm an optimistic kind of guy
;-) )

Thoughts?

--
Augustin Trancart - Oslandia
[hidden email]


_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev




--

Cordialement,
-- 
Thomas Broyer
Atol Conseils et Développements


_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
Yes, agree with the idea of removing other big obsolete files as well.

I'm not too worried about pull/pushes. If we require the files to be removed
before merge, these commits would become unreachable after the force-push, and
would eventually be garbage collected (providing we are careful at removing old
branches).

Augustin Trancart - Oslandia
[hidden email]

On 29/06/2017 12:03, Thomas Broyer wrote:

> 15 months ago (or so), big files were added to the project and then removed, and
> I also floated the idea of a history rewrite. I never got the "go" so it hasn't
> been done. Iff we go that road for the video file, I suggest we also get rid of
> those old commits.
>
> I don't have a strong opinion whether this should be done, but it's clear that
> it's now or never.
>
> I'd suggest we eventually find a way to somehow reject pull requests (we
> apparently can't reject commits/pushes) with big files (or many smaller files!);
> possibly an additional check in Travis, I don't know.
>
> 2017-06-29 11:32 GMT+02:00 Augustin Trancart <[hidden email]
> <mailto:[hidden email]>>:
>
>     Hi there!
>
>     Quick question with the release in sight: a video of 24M has been added (in
>     aabc913) and removed later (in b321c01) in the repository. This makes the clone
>     of the repository quite long even though the video is not in master any more.
>     Should we rewrite the history to get rid of it?
>
>     It's possible, but it implies a force-push on master. It means that everybody
>     will need to use "git rebase --onto master <commit-range>" for all their
>     branches after this push is done and force-push them (and maybe a git reset
>     --hard on their local master, I'm not sure how clever git is nowadays).
>
>     It's a low risk process (we'll keep a copy of the old master around), but it's a
>     bit of a hassle and requires synchronization between us. I believe it is
>     possible, but if we want to do it, we'd better do it now, before success hits us
>     hard and we have thousands of contributors (yeah, I'm an optimistic kind of guy
>     ;-) )
>
>     Thoughts?
>
>     --
>     Augustin Trancart - Oslandia
>     [hidden email] <mailto:[hidden email]>
>
>
>     _______________________________________________
>     iTowns-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.osgeo.org/mailman/listinfo/itowns-dev
>     <https://lists.osgeo.org/mailman/listinfo/itowns-dev>
>
>
>
>
> --
>
> Cordialement,
> --
> Thomas Broyer
> Atol Conseils et Développements
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Thomas Broyer


2017-06-29 12:13 GMT+02:00 Augustin Trancart <[hidden email]>:
Yes, agree with the idea of removing other big obsolete files as well.

I'm not too worried about pull/pushes. If we require the files to be removed
before merge, these commits would become unreachable after the force-push, and
would eventually be garbage collected (providing we are careful at removing old
branches).

Not sure I understand that last paragraph.
My remark wrt rejecting pull requests (or pushes) with big files was to prevent this situation from happening again (in other words, as you said, "require the files to be removed before merge", but automate it).

Note also that GitHub will keep the files anyway in their special refs/pull/ references [1], you'll just never sync them (unless you want it) because (by default) Git only syncs refs/heads/ and refs/tags/

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Vincent Picavet (ml)
In reply to this post by Pierre-Eric Pelloux-Prayer
Hello,

+1 also on history rewrite.
Try to gather all big files, not just the video.
Doing this just before the release would probably minimize the number of
branches concerned by this change ( aka : "merge now or expect trouble"
;-) .

Vincent

On 29/06/2017 11:53, Pierre-Eric Pelloux-Prayer wrote:

> +1 let's do this before the first release.
>
>
> Le 29/06/2017 à 11:32, Augustin Trancart a écrit :
>> Hi there!
>>
>> Quick question with the release in sight: a video of 24M has been added (in
>> aabc913) and removed later (in b321c01) in the repository. This makes the clone
>> of the repository quite long even though the video is not in master any more.
>> Should we rewrite the history to get rid of it?
>>
>> It's possible, but it implies a force-push on master. It means that everybody
>> will need to use "git rebase --onto master <commit-range>" for all their
>> branches after this push is done and force-push them (and maybe a git reset
>> --hard on their local master, I'm not sure how clever git is nowadays).
>>
>> It's a low risk process (we'll keep a copy of the old master around), but it's a
>> bit of a hassle and requires synchronization between us. I believe it is
>> possible, but if we want to do it, we'd better do it now, before success hits us
>> hard and we have thousands of contributors (yeah, I'm an optimistic kind of guy
>> ;-) )
>>
>> Thoughts?
>>
>>
>>
>> _______________________________________________
>> iTowns-dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
In reply to this post by Thomas Broyer
I basically trust maintainer not to merge big files, but yes, if we can automate
it, it's even better.

Augustin Trancart - Oslandia
[hidden email]

On 29/06/2017 12:34, Thomas Broyer wrote:

>
>
> 2017-06-29 12:13 GMT+02:00 Augustin Trancart <[hidden email]
> <mailto:[hidden email]>>:
>
>     Yes, agree with the idea of removing other big obsolete files as well.
>
>     I'm not too worried about pull/pushes. If we require the files to be removed
>     before merge, these commits would become unreachable after the force-push, and
>     would eventually be garbage collected (providing we are careful at removing old
>     branches).
>
>
> Not sure I understand that last paragraph.
> My remark wrt rejecting pull requests (or pushes) with big files was to prevent
> this situation from happening again (in other words, as you said, "require the
> files to be removed before merge", but automate it).
>
> Note also that GitHub will keep the files anyway in their special refs/pull/
> references [1], you'll just never sync them (unless you want it) because (by
> default) Git only syncs refs/heads/ and refs/tags/
> [1] https://help.github.com/articles/checking-out-pull-requests-locally/

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Thomas Broyer


2017-06-29 13:51 GMT+02:00 Augustin Trancart <[hidden email]>:
I basically trust maintainer not to merge big files,

Oh, you mean like for that video file? :troll:
 
but yes, if we can automate it, it's even better.

Note that apparently aabc913 didn't went through a pull request, so we couldn't have prevented it anyway (Travis could have failed, and then a push-force would have been needed anyway)

Cordialement,
-- 
Thomas Broyer
Atol Conseils et Développements


_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
Ok, so I continued my investigation, and apart from this video, the size of the
repo is also big because of the dist folder we commit to gh-pages at each
merge... I'm pretty sure that does count more than the video itself (which
shouldn't take more than it's compressed raw size, as it has not been changed
since its appearance). And I don't really know how to workaround that.

So I'm not sure it is worthy to rewrite the entire history for gaining just 20mb
(max) of our 110mb repo any more. Maybe we should just live with it, as only the
first clone is slow.

Suggestions?

Augustin Trancart - Oslandia
[hidden email]

On 29/06/2017 14:26, Thomas Broyer wrote:

>
>
> 2017-06-29 13:51 GMT+02:00 Augustin Trancart <[hidden email]
> <mailto:[hidden email]>>:
>
>     I basically trust maintainer not to merge big files,
>
>
> Oh, you mean like for that video file? :troll:
>  
>
>     but yes, if we can automate it, it's even better.
>
>
> Note that apparently aabc913 didn't went through a pull request, so we couldn't
> have prevented it anyway (Travis could have failed, and then a push-force would
> have been needed anyway)
>
> Cordialement,
> --
> Thomas Broyer
> Atol Conseils et Développements
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
Ok so, I intent to drastically reduce the size of the repo by removing from its
history the branches:
- gh-pages (only when deploy to itowns.github.io will work)
- gh-master

and the files:
- dist/* (will be partly done by removal of gh-pages)
- build/*
- data/*
- resources/*
- doc/*
- src/Core/Commander/Providers/Potree/workers.js (and actually maybe Potree/
directly)
- src/ThirdParty
- examples/gpx/ULTRA2009.gpx
- API_Doc/ (should be done by removal of gh-pages)

This implies a force-push on *all* the branches. Therefore, it is advised that
you *clone the repo back once it's done*. You'll probably need to resubmit your
PR. I can help for the rebase if necessary.

I'll keep a copy of the repo just in case :-)

Schedule: TOMORROW!

If you see something wrong in this, please raise your hand !

Augustin Trancart - Oslandia
[hidden email]

PS: FYI, when aggressively testing locally, I reduce the repo size from 103Mb to
3.9MB.

On 29/06/2017 15:56, Augustin Trancart wrote:

> Ok, so I continued my investigation, and apart from this video, the size of the
> repo is also big because of the dist folder we commit to gh-pages at each
> merge... I'm pretty sure that does count more than the video itself (which
> shouldn't take more than it's compressed raw size, as it has not been changed
> since its appearance). And I don't really know how to workaround that.
>
> So I'm not sure it is worthy to rewrite the entire history for gaining just 20mb
> (max) of our 110mb repo any more. Maybe we should just live with it, as only the
> first clone is slow.
>
> Suggestions?
>
> Augustin Trancart - Oslandia
> [hidden email]
>
> On 29/06/2017 14:26, Thomas Broyer wrote:
>>
>>
>> 2017-06-29 13:51 GMT+02:00 Augustin Trancart <[hidden email]
>> <mailto:[hidden email]>>:
>>
>>     I basically trust maintainer not to merge big files,
>>
>>
>> Oh, you mean like for that video file? :troll:
>>  
>>
>>     but yes, if we can automate it, it's even better.
>>
>>
>> Note that apparently aabc913 didn't went through a pull request, so we couldn't
>> have prevented it anyway (Travis could have failed, and then a push-force would
>> have been needed anyway)
>>
>> Cordialement,
>> --
>> Thomas Broyer
>> Atol Conseils et Développements
>>
>
>
>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
To be more precise: a prerequisite for this is to manage to upload our docs to
itowns.github.io. Pepp and I are currently working on this.

Augustin Trancart - Oslandia
[hidden email]

On 03/07/2017 19:06, Augustin Trancart wrote:

> Ok so, I intent to drastically reduce the size of the repo by removing from its
> history the branches:
> - gh-pages (only when deploy to itowns.github.io will work)
> - gh-master
>
> and the files:
> - dist/* (will be partly done by removal of gh-pages)
> - build/*
> - data/*
> - resources/*
> - doc/*
> - src/Core/Commander/Providers/Potree/workers.js (and actually maybe Potree/
> directly)
> - src/ThirdParty
> - examples/gpx/ULTRA2009.gpx
> - API_Doc/ (should be done by removal of gh-pages)
>
> This implies a force-push on *all* the branches. Therefore, it is advised that
> you *clone the repo back once it's done*. You'll probably need to resubmit your
> PR. I can help for the rebase if necessary.
>
> I'll keep a copy of the repo just in case :-)
>
> Schedule: TOMORROW!
>
> If you see something wrong in this, please raise your hand !
>
> Augustin Trancart - Oslandia
> [hidden email]
>
> PS: FYI, when aggressively testing locally, I reduce the repo size from 103Mb to
> 3.9MB.
>
> On 29/06/2017 15:56, Augustin Trancart wrote:
>> Ok, so I continued my investigation, and apart from this video, the size of the
>> repo is also big because of the dist folder we commit to gh-pages at each
>> merge... I'm pretty sure that does count more than the video itself (which
>> shouldn't take more than it's compressed raw size, as it has not been changed
>> since its appearance). And I don't really know how to workaround that.
>>
>> So I'm not sure it is worthy to rewrite the entire history for gaining just 20mb
>> (max) of our 110mb repo any more. Maybe we should just live with it, as only the
>> first clone is slow.
>>
>> Suggestions?
>>
>> Augustin Trancart - Oslandia
>> [hidden email]
>>
>> On 29/06/2017 14:26, Thomas Broyer wrote:
>>>
>>>
>>> 2017-06-29 13:51 GMT+02:00 Augustin Trancart <[hidden email]
>>> <mailto:[hidden email]>>:
>>>
>>>     I basically trust maintainer not to merge big files,
>>>
>>>
>>> Oh, you mean like for that video file? :troll:
>>>  
>>>
>>>     but yes, if we can automate it, it's even better.
>>>
>>>
>>> Note that apparently aabc913 didn't went through a pull request, so we couldn't
>>> have prevented it anyway (Travis could have failed, and then a push-force would
>>> have been needed anyway)
>>>
>>> Cordialement,
>>> --
>>> Thomas Broyer
>>> Atol Conseils et Développements
>>>
>>
>>
>>
>> _______________________________________________
>> iTowns-dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>>
>
>
>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Thomas Broyer
Instead of removing gh-pages altogether (unless there's a good reason to do so), couldn't we change the auto-deploy script to force-push?

Otherwise sounds good (the old files I recalled of were in resources/ https://github.com/iTowns/itowns2/commit/95bf265e33923c0a8e45b06558160e76f414e00d)

Le 3 juil. 2017 7:13 PM, "Augustin Trancart" <[hidden email]> a écrit :
To be more precise: a prerequisite for this is to manage to upload our docs to
itowns.github.io. Pepp and I are currently working on this.

Augustin Trancart - Oslandia
[hidden email]

On 03/07/2017 19:06, Augustin Trancart wrote:
> Ok so, I intent to drastically reduce the size of the repo by removing from its
> history the branches:
> - gh-pages (only when deploy to itowns.github.io will work)
> - gh-master
>
> and the files:
> - dist/* (will be partly done by removal of gh-pages)
> - build/*
> - data/*
> - resources/*
> - doc/*
> - src/Core/Commander/Providers/Potree/workers.js (and actually maybe Potree/
> directly)
> - src/ThirdParty
> - examples/gpx/ULTRA2009.gpx
> - API_Doc/ (should be done by removal of gh-pages)
>
> This implies a force-push on *all* the branches. Therefore, it is advised that
> you *clone the repo back once it's done*. You'll probably need to resubmit your
> PR. I can help for the rebase if necessary.
>
> I'll keep a copy of the repo just in case :-)
>
> Schedule: TOMORROW!
>
> If you see something wrong in this, please raise your hand !
>
> Augustin Trancart - Oslandia
> [hidden email]
>
> PS: FYI, when aggressively testing locally, I reduce the repo size from 103Mb to
> 3.9MB.
>
> On 29/06/2017 15:56, Augustin Trancart wrote:
>> Ok, so I continued my investigation, and apart from this video, the size of the
>> repo is also big because of the dist folder we commit to gh-pages at each
>> merge... I'm pretty sure that does count more than the video itself (which
>> shouldn't take more than it's compressed raw size, as it has not been changed
>> since its appearance). And I don't really know how to workaround that.
>>
>> So I'm not sure it is worthy to rewrite the entire history for gaining just 20mb
>> (max) of our 110mb repo any more. Maybe we should just live with it, as only the
>> first clone is slow.
>>
>> Suggestions?
>>
>> Augustin Trancart - Oslandia
>> [hidden email]
>>
>> On 29/06/2017 14:26, Thomas Broyer wrote:
>>>
>>>
>>> 2017-06-29 13:51 GMT+02:00 Augustin Trancart <[hidden email]
>>> <mailto:[hidden email]>>:
>>>
>>>     I basically trust maintainer not to merge big files,
>>>
>>>
>>> Oh, you mean like for that video file? :troll:
>>>
>>>
>>>     but yes, if we can automate it, it's even better.
>>>
>>>
>>> Note that apparently aabc913 didn't went through a pull request, so we couldn't
>>> have prevented it anyway (Travis could have failed, and then a push-force would
>>> have been needed anyway)
>>>
>>> Cordialement,
>>> --
>>> Thomas Broyer
>>> Atol Conseils et Développements
>>>
>>
>>
>>
>> _______________________________________________
>> iTowns-dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>>
>
>
>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>


_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev



_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
You mean to never have more than one commit in gh-pages? It is a solution too, yes.

We thought about pushing all our docs directly to itowns.github.io, removing the
need of gh-pages branch. We don't really care if this repo is slow to clone (as
few people will ever do it), and we'll keep the doc history that way. WDYT?

Augustin Trancart - Oslandia
[hidden email]

On 03/07/2017 19:35, Thomas Broyer wrote:

> Instead of removing gh-pages altogether (unless there's a good reason to do so),
> couldn't we change the auto-deploy script to force-push?
>
> Otherwise sounds good (the old files I recalled of were in
> resources/ https://github.com/iTowns/itowns2/commit/95bf265e33923c0a8e45b06558160e76f414e00d)
>
> Le 3 juil. 2017 7:13 PM, "Augustin Trancart" <[hidden email]
> <mailto:[hidden email]>> a écrit :
>
>     To be more precise: a prerequisite for this is to manage to upload our docs to
>     itowns.github.io <http://itowns.github.io>. Pepp and I are currently working
>     on this.
>
>     Augustin Trancart - Oslandia
>     [hidden email] <mailto:[hidden email]>
>
>     On 03/07/2017 19:06, Augustin Trancart wrote:
>     > Ok so, I intent to drastically reduce the size of the repo by removing
>     from its
>     > history the branches:
>     > - gh-pages (only when deploy to itowns.github.io <http://itowns.github.io>
>     will work)
>     > - gh-master
>     >
>     > and the files:
>     > - dist/* (will be partly done by removal of gh-pages)
>     > - build/*
>     > - data/*
>     > - resources/*
>     > - doc/*
>     > - src/Core/Commander/Providers/Potree/workers.js (and actually maybe Potree/
>     > directly)
>     > - src/ThirdParty
>     > - examples/gpx/ULTRA2009.gpx
>     > - API_Doc/ (should be done by removal of gh-pages)
>     >
>     > This implies a force-push on *all* the branches. Therefore, it is advised that
>     > you *clone the repo back once it's done*. You'll probably need to resubmit
>     your
>     > PR. I can help for the rebase if necessary.
>     >
>     > I'll keep a copy of the repo just in case :-)
>     >
>     > Schedule: TOMORROW!
>     >
>     > If you see something wrong in this, please raise your hand !
>     >
>     > Augustin Trancart - Oslandia
>     > [hidden email] <mailto:[hidden email]>
>     >
>     > PS: FYI, when aggressively testing locally, I reduce the repo size from
>     103Mb to
>     > 3.9MB.
>     >
>     > On 29/06/2017 15:56, Augustin Trancart wrote:
>     >> Ok, so I continued my investigation, and apart from this video, the size
>     of the
>     >> repo is also big because of the dist folder we commit to gh-pages at each
>     >> merge... I'm pretty sure that does count more than the video itself (which
>     >> shouldn't take more than it's compressed raw size, as it has not been changed
>     >> since its appearance). And I don't really know how to workaround that.
>     >>
>     >> So I'm not sure it is worthy to rewrite the entire history for gaining
>     just 20mb
>     >> (max) of our 110mb repo any more. Maybe we should just live with it, as
>     only the
>     >> first clone is slow.
>     >>
>     >> Suggestions?
>     >>
>     >> Augustin Trancart - Oslandia
>     >> [hidden email] <mailto:[hidden email]>
>     >>
>     >> On 29/06/2017 14:26, Thomas Broyer wrote:
>     >>>
>     >>>
>     >>> 2017-06-29 13:51 GMT+02:00 Augustin Trancart
>     <[hidden email] <mailto:[hidden email]>
>     >>> <mailto:[hidden email]
>     <mailto:[hidden email]>>>:
>     >>>
>     >>>     I basically trust maintainer not to merge big files,
>     >>>
>     >>>
>     >>> Oh, you mean like for that video file? :troll:
>     >>>
>     >>>
>     >>>     but yes, if we can automate it, it's even better.
>     >>>
>     >>>
>     >>> Note that apparently aabc913 didn't went through a pull request, so we
>     couldn't
>     >>> have prevented it anyway (Travis could have failed, and then a
>     push-force would
>     >>> have been needed anyway)
>     >>>
>     >>> Cordialement,
>     >>> --
>     >>> Thomas Broyer
>     >>> Atol Conseils et Développements
>     >>>
>     >>
>     >>
>     >>
>     >> _______________________________________________
>     >> iTowns-dev mailing list
>     >> [hidden email] <mailto:[hidden email]>
>     >> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>     <https://lists.osgeo.org/mailman/listinfo/itowns-dev>
>     >>
>     >
>     >
>     >
>     > _______________________________________________
>     > iTowns-dev mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://lists.osgeo.org/mailman/listinfo/itowns-dev
>     <https://lists.osgeo.org/mailman/listinfo/itowns-dev>
>     >
>
>
>     _______________________________________________
>     iTowns-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.osgeo.org/mailman/listinfo/itowns-dev
>     <https://lists.osgeo.org/mailman/listinfo/itowns-dev>
>
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Thomas Broyer
If itowns.github.io contains manually-maintained files, then if we want good documentation we will care about clone performance.

But yes, that's what I meant (only one commit in gh-pages)

Le 3 juil. 2017 7:41 PM, "Augustin Trancart" <[hidden email]> a écrit :
You mean to never have more than one commit in gh-pages? It is a solution too, yes.

We thought about pushing all our docs directly to itowns.github.io, removing the
need of gh-pages branch. We don't really care if this repo is slow to clone (as
few people will ever do it), and we'll keep the doc history that way. WDYT?

Augustin Trancart - Oslandia
[hidden email]

On 03/07/2017 19:35, Thomas Broyer wrote:
> Instead of removing gh-pages altogether (unless there's a good reason to do so),
> couldn't we change the auto-deploy script to force-push?
>
> Otherwise sounds good (the old files I recalled of were in
> resources/ https://github.com/iTowns/itowns2/commit/95bf265e33923c0a8e45b06558160e76f414e00d)
>
> Le 3 juil. 2017 7:13 PM, "Augustin Trancart" <[hidden email]
> <mailto:[hidden email]>> a écrit :
>
>     To be more precise: a prerequisite for this is to manage to upload our docs to
>     itowns.github.io <http://itowns.github.io>. Pepp and I are currently working
>     on this.
>
>     Augustin Trancart - Oslandia
>     [hidden email] <mailto:[hidden email]>
>
>     On 03/07/2017 19:06, Augustin Trancart wrote:
>     > Ok so, I intent to drastically reduce the size of the repo by removing
>     from its
>     > history the branches:
>     > - gh-pages (only when deploy to itowns.github.io <http://itowns.github.io>
>     will work)
>     > - gh-master
>     >
>     > and the files:
>     > - dist/* (will be partly done by removal of gh-pages)
>     > - build/*
>     > - data/*
>     > - resources/*
>     > - doc/*
>     > - src/Core/Commander/Providers/Potree/workers.js (and actually maybe Potree/
>     > directly)
>     > - src/ThirdParty
>     > - examples/gpx/ULTRA2009.gpx
>     > - API_Doc/ (should be done by removal of gh-pages)
>     >
>     > This implies a force-push on *all* the branches. Therefore, it is advised that
>     > you *clone the repo back once it's done*. You'll probably need to resubmit
>     your
>     > PR. I can help for the rebase if necessary.
>     >
>     > I'll keep a copy of the repo just in case :-)
>     >
>     > Schedule: TOMORROW!
>     >
>     > If you see something wrong in this, please raise your hand !
>     >
>     > Augustin Trancart - Oslandia
>     > [hidden email] <mailto:[hidden email]>
>     >
>     > PS: FYI, when aggressively testing locally, I reduce the repo size from
>     103Mb to
>     > 3.9MB.
>     >
>     > On 29/06/2017 15:56, Augustin Trancart wrote:
>     >> Ok, so I continued my investigation, and apart from this video, the size
>     of the
>     >> repo is also big because of the dist folder we commit to gh-pages at each
>     >> merge... I'm pretty sure that does count more than the video itself (which
>     >> shouldn't take more than it's compressed raw size, as it has not been changed
>     >> since its appearance). And I don't really know how to workaround that.
>     >>
>     >> So I'm not sure it is worthy to rewrite the entire history for gaining
>     just 20mb
>     >> (max) of our 110mb repo any more. Maybe we should just live with it, as
>     only the
>     >> first clone is slow.
>     >>
>     >> Suggestions?
>     >>
>     >> Augustin Trancart - Oslandia
>     >> [hidden email] <mailto:[hidden email]>
>     >>
>     >> On 29/06/2017 14:26, Thomas Broyer wrote:
>     >>>
>     >>>
>     >>> 2017-06-29 13:51 GMT+02:00 Augustin Trancart
>     <[hidden email] <mailto:[hidden email]>
>     >>> <mailto:[hidden email]
>     <mailto:[hidden email]>>>:
>     >>>
>     >>>     I basically trust maintainer not to merge big files,
>     >>>
>     >>>
>     >>> Oh, you mean like for that video file? :troll:
>     >>>
>     >>>
>     >>>     but yes, if we can automate it, it's even better.
>     >>>
>     >>>
>     >>> Note that apparently aabc913 didn't went through a pull request, so we
>     couldn't
>     >>> have prevented it anyway (Travis could have failed, and then a
>     push-force would
>     >>> have been needed anyway)
>     >>>
>     >>> Cordialement,
>     >>> --
>     >>> Thomas Broyer
>     >>> Atol Conseils et Développements
>     >>>
>     >>
>     >>
>     >>
>     >> _______________________________________________
>     >> iTowns-dev mailing list
>     >> [hidden email] <mailto:[hidden email]>
>     >> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>     <https://lists.osgeo.org/mailman/listinfo/itowns-dev>
>     >>
>     >
>     >
>     >
>     > _______________________________________________
>     > iTowns-dev mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://lists.osgeo.org/mailman/listinfo/itowns-dev
>     <https://lists.osgeo.org/mailman/listinfo/itowns-dev>
>     >
>
>
>     _______________________________________________
>     iTowns-dev mailing list


_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev
Reply | Threaded
Open this post in threaded view
|

Re: Video in repo

Augustin Trancart
In reply to this post by Augustin Trancart
Starting the clean now. Please don't push :-)


Augustin Trancart - Oslandia
[hidden email]

On 03/07/2017 19:13, Augustin Trancart wrote:

> To be more precise: a prerequisite for this is to manage to upload our docs to
> itowns.github.io. Pepp and I are currently working on this.
>
> Augustin Trancart - Oslandia
> [hidden email]
>
> On 03/07/2017 19:06, Augustin Trancart wrote:
>> Ok so, I intent to drastically reduce the size of the repo by removing from its
>> history the branches:
>> - gh-pages (only when deploy to itowns.github.io will work)
>> - gh-master
>>
>> and the files:
>> - dist/* (will be partly done by removal of gh-pages)
>> - build/*
>> - data/*
>> - resources/*
>> - doc/*
>> - src/Core/Commander/Providers/Potree/workers.js (and actually maybe Potree/
>> directly)
>> - src/ThirdParty
>> - examples/gpx/ULTRA2009.gpx
>> - API_Doc/ (should be done by removal of gh-pages)
>>
>> This implies a force-push on *all* the branches. Therefore, it is advised that
>> you *clone the repo back once it's done*. You'll probably need to resubmit your
>> PR. I can help for the rebase if necessary.
>>
>> I'll keep a copy of the repo just in case :-)
>>
>> Schedule: TOMORROW!
>>
>> If you see something wrong in this, please raise your hand !
>>
>> Augustin Trancart - Oslandia
>> [hidden email]
>>
>> PS: FYI, when aggressively testing locally, I reduce the repo size from 103Mb to
>> 3.9MB.
>>
>> On 29/06/2017 15:56, Augustin Trancart wrote:
>>> Ok, so I continued my investigation, and apart from this video, the size of the
>>> repo is also big because of the dist folder we commit to gh-pages at each
>>> merge... I'm pretty sure that does count more than the video itself (which
>>> shouldn't take more than it's compressed raw size, as it has not been changed
>>> since its appearance). And I don't really know how to workaround that.
>>>
>>> So I'm not sure it is worthy to rewrite the entire history for gaining just 20mb
>>> (max) of our 110mb repo any more. Maybe we should just live with it, as only the
>>> first clone is slow.
>>>
>>> Suggestions?
>>>
>>> Augustin Trancart - Oslandia
>>> [hidden email]
>>>
>>> On 29/06/2017 14:26, Thomas Broyer wrote:
>>>>
>>>>
>>>> 2017-06-29 13:51 GMT+02:00 Augustin Trancart <[hidden email]
>>>> <mailto:[hidden email]>>:
>>>>
>>>>     I basically trust maintainer not to merge big files,
>>>>
>>>>
>>>> Oh, you mean like for that video file? :troll:
>>>>  
>>>>
>>>>     but yes, if we can automate it, it's even better.
>>>>
>>>>
>>>> Note that apparently aabc913 didn't went through a pull request, so we couldn't
>>>> have prevented it anyway (Travis could have failed, and then a push-force would
>>>> have been needed anyway)
>>>>
>>>> Cordialement,
>>>> --
>>>> Thomas Broyer
>>>> Atol Conseils et Développements
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> iTowns-dev mailing list
>>> [hidden email]
>>> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>>>
>>
>>
>>
>> _______________________________________________
>> iTowns-dev mailing list
>> [hidden email]
>> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>>
>
>
>
> _______________________________________________
> iTowns-dev mailing list
> [hidden email]
> https://lists.osgeo.org/mailman/listinfo/itowns-dev
>

_______________________________________________
iTowns-dev mailing list
[hidden email]
https://lists.osgeo.org/mailman/listinfo/itowns-dev

signature.asc (836 bytes) Download Attachment