81 Commits

Author SHA1 Message Date
wxiaoguang
4e2f1ee58d Refactor web package and context package (#25298)
1. The "web" package shouldn't depends on "modules/context" package,
instead, let each "web context" register themselves to the "web"
package.
2. The old Init/Free doesn't make sense, so simplify it
* The ctx in "Init(ctx)" is never used, and shouldn't be used that way
* The "Free" is never called and shouldn't be called because the SSPI
instance is shared

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-06-18 09:59:09 +02:00
KN4CK3R
cdb088cec2 Add CRAN package registry (#22331)
This PR adds a [CRAN](https://cran.r-project.org/) package registry.

![grafik](https://user-images.githubusercontent.com/1666336/210450039-d6fa6f77-20cd-4741-89a8-1624def267f7.png)
2023-05-22 10:57:49 +08:00
wxiaoguang
6b33152b7d Decouple the different contexts from each other (#24786)
Replace #16455

Close #21803

Mixing different Gitea contexts together causes some problems:

1. Unable to respond proper content when error occurs, eg: Web should
respond HTML while API should respond JSON
2. Unclear dependency, eg: it's unclear when Context is used in
APIContext, which fields should be initialized, which methods are
necessary.


To make things clear, this PR introduces a Base context, it only
provides basic Req/Resp/Data features.

This PR mainly moves code. There are still many legacy problems and
TODOs in code, leave unrelated changes to future PRs.
2023-05-21 09:50:53 +08:00
KN4CK3R
5968c63a11 Add Go package registry (#24687)
Fixes #7608

This PR adds a Go package registry usable with the Go proxy protocol.

![grafik](https://github.com/go-gitea/gitea/assets/1666336/328feb5c-3df2-4f9d-8eae-fe3126d14c37)
2023-05-14 23:38:40 +08:00
KN4CK3R
9173e079ae Add Alpine package registry (#23714)
This PR adds an Alpine package registry. You can follow [this
tutorial](https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package)
to build a *.apk package for testing.

This functionality is similar to the Debian registry (#22854) and
therefore shares some methods. I marked this PR as blocked because it
should be merged after #22854.


![grafik](https://user-images.githubusercontent.com/1666336/227779595-b76163aa-eea1-4a79-9583-775c24ad74e8.png)

---------

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-12 17:27:50 +00:00
KN4CK3R
05209f0d1d Add RPM registry (#23380)
Fixes #20751

This PR adds a RPM package registry. You can follow [this
tutorial](https://opensource.com/article/18/9/how-build-rpm-packages) to
build a *.rpm package for testing.

This functionality is similar to the Debian registry (#22854) and
therefore shares some methods. I marked this PR as blocked because it
should be merged after #22854.


![grafik](https://user-images.githubusercontent.com/1666336/223806549-d8784fd9-9d79-46a2-9ae2-f038594f636a.png)
2023-05-05 20:33:37 +00:00
KN4CK3R
723598b803 Implement Cargo HTTP index (#24452)
This implements the HTTP index
[RFC](https://rust-lang.github.io/rfcs/2789-sparse-index.html) for Cargo
registries.

Currently this is a preview feature and you need to use the nightly of
`cargo`:

`cargo +nightly -Z sparse-registry update`

See https://github.com/rust-lang/cargo/issues/9069 for more information.

---------

Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-03 16:58:43 -04:00
KN4CK3R
bf999e4069 Add Debian package registry (#24426)
Co-authored-by: @awkwardbunny

This PR adds a Debian package registry.
You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing.
Source packages are not supported at the moment and I did not find
documentation of the architecture "all" and how these packages should be
treated.


![grafik](https://user-images.githubusercontent.com/1666336/218126879-eb80a866-775c-4c8e-8529-5797203a64e6.png)

Part of #20751.

Revised copy of #22854.

---------

Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Giteabot <teabot@gitea.io>
2023-05-02 12:31:35 -04:00
Yarden Shoham
c0ddec8a2a Revert "Add Debian package registry" (#24412)
Reverts go-gitea/gitea#22854
2023-04-28 18:06:41 -04:00
KN4CK3R
bf77e2163b Add Debian package registry (#22854)
Co-authored-by: @awkwardbunny

This PR adds a Debian package registry. You can follow [this
tutorial](https://www.baeldung.com/linux/create-debian-package) to build
a *.deb package for testing. Source packages are not supported at the
moment and I did not find documentation of the architecture "all" and
how these packages should be treated.

---------

Co-authored-by: Brian Hong <brian@hongs.me>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2023-04-28 17:51:36 -04:00
John Olheiser
5e36024105 Require repo scope for PATs for private repos and basic authentication (#24362)
> The scoped token PR just checked all API routes but in fact, some web
routes like `LFS`, git `HTTP`, container, and attachments supports basic
auth. This PR added scoped token check for them.

---------

Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-04-26 19:24:03 -05:00
wxiaoguang
b9a97ccd0e Refactor web route (#24080)
The old code is unnecessarily complex, and has many misuses.

Old code "wraps" a lot, wrap wrap wrap, it's difficult to understand
which kind of handler is used.

The new code uses a general approach, we do not need to write all kinds
of handlers into the "wrapper", do not need to wrap them again and
again.

New code, there are only 2 concepts:

1. HandlerProvider: `func (h any) (handlerProvider func (next)
http.Handler)`, it can be used as middleware
2. Use HandlerProvider to get the final HandlerFunc, and use it for
`r.Get()`


And we can decouple the route package from context package (see the
TODO).

# FAQ

## Is `reflect` safe?

Yes, all handlers are checked during startup, see the `preCheckHandler`
comment. If any handler is wrong, developers could know it in the first
time.

## Does `reflect` affect performance?

No. https://github.com/go-gitea/gitea/pull/24080#discussion_r1164825901

1. This reflect code only runs for each web handler call, handler is far
more slower: 10ms-50ms
2. The reflect is pretty fast (comparing to other code): 0.000265ms
3. XORM has more reflect operations already
2023-04-20 14:49:06 -04:00
yp05327
bb6c670cff Add actions support to package auth verification (#23729)
Partly fixes https://github.com/go-gitea/gitea/issues/23642

Error info:

![image](https://user-images.githubusercontent.com/18380374/227827027-4280a368-ec9e-49e0-bb93-6b496ada7cd9.png)
ActionsUser (userID -2) is used to login in to docker in action jobs.

Due to we have no permission policy settings of ActionsUser now,
ActionsUser can only access public registry by this quick fix.
2023-04-10 15:21:03 +08:00
KN4CK3R
fbd4eaceed Display image size for multiarch container images (#23821)
Fixes #23771

Changes the display of different architectures for multiarch images to
show the image size:

![grafik](https://user-images.githubusercontent.com/1666336/228781477-cc76c4d1-4728-434f-8a27-fc008790d924.png)
2023-04-02 17:53:37 +08:00
Mai-Lapyst
5cd1d6c93b Set repository link based on the url in package.json for npm packages (#20379)
automatically set repository link for package based on the repository
url present inside package.json

closes #20146
2023-03-28 13:55:03 -04:00
Samuel FORESTIER
dbdb5ba33e Sort Python package descriptors by version to mimic PyPI format (#23550)
---

Hi, very naive and **untested** first time ever Go code, feel free to
reject/edit this as needed.
(PyPI actually performs "naive" string comparison too)

---------

Co-authored-by: Lauris BH <lauris@nix.lv>
2023-03-19 15:19:37 +01:00
Zettat123
46addc1f93 Return repository in npm package metadata endpoint (#23539)
Close #23444 

Add `Repository` to npm package `Metadata` struct so the `repository` in
`package.json` can be stored and be returned in the endpoint.

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2023-03-17 14:39:19 -04:00
KN4CK3R
8570593d55 Add package registry architecture overview (#23445)
As announced in #22810 I added a readme file to help understanding how
the package registry archictecture works and how the go packages are
related.
2023-03-13 18:15:09 -04:00
KN4CK3R
c709fa17a7 Add Swift package registry (#22404)
This PR adds a [Swift](https://www.swift.org/) package registry.


![grafik](https://user-images.githubusercontent.com/1666336/211842523-07521cbd-8fb6-400f-820c-ee8048b05ae8.png)
2023-03-13 15:28:39 -05:00
zeripath
1319ba6742 Use minio/sha256-simd for accelerated SHA256 (#23052)
minio/sha256-simd provides additional acceleration for SHA256 using
AVX512, SHA Extensions for x86 and ARM64 for ARM.

It provides a drop-in replacement for crypto/sha256 and if the
extensions are not available it falls back to standard crypto/sha256.

---------

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2023-02-22 14:21:46 -05:00
KN4CK3R
007d181bb5 Notify on container image create (#22806)
Fixes #22791

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-18 13:36:38 +08:00
KN4CK3R
9057a008a1 Add /$count endpoints for NuGet v2 (#22855)
Fixes #22838

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-11 19:30:44 +08:00
KN4CK3R
f8c1e14a13 Use import of OCI structs (#22765)
Fixes #22758

Otherwise we would need to rewrite the structs in `oci.go`.
2023-02-06 10:07:09 +00:00
KN4CK3R
d987ac6bf1 Add Chef package registry (#22554)
This PR implements a [Chef registry](https://chef.io/) to manage
cookbooks. This package type was a bit complicated because Chef uses RSA
signed requests as authentication with the registry.


![grafik](https://user-images.githubusercontent.com/1666336/213747995-46819fd8-c3d6-45a2-afd4-a4c3c8505a4a.png)


![grafik](https://user-images.githubusercontent.com/1666336/213748145-d01c9e81-d4dd-41e3-a3cc-8241862c3166.png)

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-06 09:49:21 +08:00
KN4CK3R
df789d962b Add Cargo package registry (#21888)
This PR implements a [Cargo registry](https://doc.rust-lang.org/cargo/)
to manage Rust packages. This package type was a little bit more
complicated because Cargo needs an additional Git repository to store
its package index.

Screenshots:

![grafik](https://user-images.githubusercontent.com/1666336/203102004-08d812ac-c066-4969-9bda-2fed818554eb.png)

![grafik](https://user-images.githubusercontent.com/1666336/203102141-d9970f14-dca6-4174-b17a-50ba1bd79087.png)

![grafik](https://user-images.githubusercontent.com/1666336/203102244-dc05743b-78b6-4d97-998e-ef76341a978f.png)

---------

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2023-02-05 18:12:31 +08:00
KN4CK3R
6ba9ff7b48 Add Conda package registry (#22262)
This PR adds a [Conda](https://conda.io/) package registry.
2023-02-01 12:30:39 -06:00
KN4CK3R
d283a31f03 Check quota limits for container uploads (#22450)
The test coverage has revealed that container packages were not checked
against the quota limits.
2023-01-29 11:34:29 -06:00
KN4CK3R
3510d7e33a Fix container blob mount (#22226) 2023-01-16 17:35:48 -05:00
KN4CK3R
3fef47b41c Use ErrInvalidArgument in packages (#22268)
Related to
https://github.com/go-gitea/gitea/pull/22262#discussion_r1059010774

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
2022-12-31 12:49:37 +01:00
KN4CK3R
a35749893b Move convert package to services (#22264)
Addition to #22256

The `convert` package relies heavily on different models which is
[disallowed by our definition of
modules](https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md#design-guideline).
This helps to prevent possible import cycles.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-12-29 10:57:15 +08:00
Lunny Xiao
ca67c5a8a7 refactor auth interface to return error when verify failure (#22119)
This PR changed the Auth interface signature from 
`Verify(http *http.Request, w http.ResponseWriter, store DataStore, sess
SessionStore) *user_model.User`
to 
`Verify(http *http.Request, w http.ResponseWriter, store DataStore, sess
SessionStore) (*user_model.User, error)`.

There is a new return argument `error` which means the verification
condition matched but verify process failed, we should stop the auth
process.

Before this PR, when return a `nil` user, we don't know the reason why
it returned `nil`. If the match condition is not satisfied or it
verified failure? For these two different results, we should have
different handler. If the match condition is not satisfied, we should
try next auth method and if there is no more auth method, it's an
anonymous user. If the condition matched but verify failed, the auth
process should be stop and return immediately.

This will fix #20563

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: Jason Song <i@wolfogre.com>
2022-12-28 13:53:28 +08:00
KN4CK3R
86ace4b5c2 Normalize NuGet package version on upload (#22186)
Fixes #22178

After this change upload versions with different semver metadata are
treated as the same version and trigger a duplicated version error.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-12-20 22:20:23 -05:00
KN4CK3R
07461e18d3 Fix condition for is_internal (#22095)
depends on #22094

Fixes https://codeberg.org/forgejo/forgejo/issues/77

The old logic did not consider `is_internal`.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2022-12-14 17:16:01 +08:00
silverwind
0585ac3ac6 Update go dev dependencies (#22064)
`golangci-lint`
[deprecated](https://github.com/golangci/golangci-lint/issues/1841) a
bunch of linters, removed them.
2022-12-08 16:21:37 +08:00
Lunny Xiao
0a7d3ff786 refactor some functions to support ctx as first parameter (#21878)
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: Lauris BH <lauris@nix.lv>
2022-12-03 10:48:26 +08:00
Jason Song
9607750b5e Replace fmt.Sprintf with hex.EncodeToString (#21960)
`hex.EncodeToString` has better performance than `fmt.Sprintf("%x",
[]byte)`, we should use it as much as possible.

I'm not an extreme fan of performance, so I think there are some
exceptions:

- `fmt.Sprintf("%x", func(...)[N]byte())`
- We can't slice the function return value directly, and it's not worth
adding lines.
    ```diff
    func A()[20]byte { ... }
    - a := fmt.Sprintf("%x", A())
    - a := hex.EncodeToString(A()[:]) // invalid
    + tmp := A()
    + a := hex.EncodeToString(tmp[:])
    ```
- `fmt.Sprintf("%X", []byte)`
- `strings.ToUpper(hex.EncodeToString(bytes))` has even worse
performance.
2022-11-28 11:19:18 +00:00
flynnnnnnnnnn
e81ccc406b Implement FSFE REUSE for golang files (#21840)
Change all license headers to comply with REUSE specification.

Fix #16132

Co-authored-by: flynnnnnnnnnn <flynnnnnnnnnn@github>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
2022-11-27 18:20:29 +00:00
KN4CK3R
a1ae83f36e Workaround for container registry push/pull errors (#21862)
This PR addresses #19586

I added a mutex to the upload version creation which will prevent the
push errors when two requests try to create these database entries. I'm
not sure if this should be the final solution for this problem.

I added a workaround to allow a reupload of missing blobs. Normally a
reupload is skipped because the database knows the blob is already
present. The workaround checks if the blob exists on the file system.
This should not be needed anymore with the above fix so I marked this
code to be removed with Gitea v1.20.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-25 13:47:46 +08:00
KN4CK3R
fc7a2d5a95 Add support for HEAD requests in Maven registry (#21834)
Related #18543

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-24 16:25:13 +02:00
Jason Song
e4eaa68a2b Replace yaml.v2 with yaml.v3 (#21832)
I don't see why we have to use two versions of yaml. The difference
between the two versions has nothing to do with our usage.
2022-11-21 16:36:59 +08:00
KN4CK3R
044c754ea5 Add context.Context to more methods (#21546)
This PR adds a context parameter to a bunch of methods. Some helper
`xxxCtx()` methods got replaced with the normal name now.

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-19 16:12:33 +08:00
KN4CK3R
43ab9324c5 Fix setting HTTP headers after write (#21833)
The headers can't be modified after it was send to the client.
2022-11-18 01:55:15 +08:00
Lunny Xiao
34283a74e8 Allow detect whether it's in a database transaction for a context.Context (#21756)
Fix #19513

This PR introduce a new db method `InTransaction(context.Context)`,
and also builtin check on `db.TxContext` and `db.WithTx`.
There is also a new method `db.AutoTx` has been introduced but could be used by other PRs.

`WithTx` will always open a new transaction, if a transaction exist in context, return an error.
`AutoTx` will try to open a new transaction if no transaction exist in context.
That means it will always enter a transaction if there is no error.

Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
2022-11-12 21:18:50 +01:00
zeripath
a0a425a13b Add some documentation to packages (#21648)
In #21637 it was mentioned that the purpose of the API routes for the
packages is unclear. This PR adds some documentation.

Fix #21637

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2022-11-12 18:59:15 +00:00
KN4CK3R
20674dd05d Add package registry quota limits (#21584)
Related #20471

This PR adds global quota limits for the package registry. Settings for
individual users/orgs can be added in a seperate PR using the settings
table.

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-09 14:34:27 +08:00
Wayne Starr
8c1d9885e5 Remove semver compatible flag and change pypi to an array of test cases (#21708)
This addresses #21707 and adds a second package test case for a
non-semver compatible version (this might be overkill though since you
could also edit the old package version to have an epoch in front and
see the error, this just seemed more flexible for the future).

Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
2022-11-08 09:41:39 +08:00
KN4CK3R
fd89c062bd Allow local package identifiers for PyPI packages (#21690)
Fixes #21683

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-11-07 21:35:04 +02:00
KN4CK3R
7c11a73833 Fix package access for admins and inactive users (#21580)
I noticed an admin is not allowed to upload packages for other users
because `ctx.IsSigned` was not set.
I added a check for `user.IsActive` and `user.ProhibitLogin` too because
both was not checked. Tests enforce this now.

Co-authored-by: Lauris BH <lauris@nix.lv>
2022-10-24 22:23:25 +03:00
Wayne Starr
49a4464160 Allow for resolution of NPM registry paths that match upstream (#21568)
This PR fixes issue #21567 allowing for package tarball URLs to match
the upstream registry (and GitLab/JFrog Artifactory URLs). It uses a
regex to parse the filename (which contains the NPM version) and does a
fuzzy search to pull it out. The regex was built/expanded from
http://json.schemastore.org/package,
https://github.com/Masterminds/semver, and
https://docs.npmjs.com/cli/v6/using-npm/semver and is testable here:
https://regex101.com/r/OydBJq/5

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2022-10-24 21:50:22 +08:00
KN4CK3R
11d3677818 Enforce grouped NuGet search results (#21442)
Fixes #21434

Added tests to enforce this behaviour.

Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
2022-10-17 01:18:09 +08:00