gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5868 commits 28 branches 278 tags 135 MiB
Commit graph

731 commits

Author SHA1 Message Date
Chris Lu cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu 1f984d2645 refactor buffer pool 2021-04-01 02:20:00 -07:00
Chris Lu ac875976c0 filer, volume: add concurrent upload size limit to avoid OOM 
add some back pressure when writes are slow
 2021-03-30 02:10:53 -07:00
Chris Lu a1e18a1384 server add memory profiling 2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision 9385e2224d fix postgres2 upsertQuery syntax also 2021-03-30 10:51:52 +03:00
Chris Lu a5662c9e5e adjust ordering 2021-03-30 00:26:57 -07:00
Chris Lu bec74c3e6a
Merge pull request #1948 from LazyDBA247-Anyvision/master 
add enableUpsert=true
 2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision 9f1cab179c use a compatible pg syntax for upsert 2021-03-30 10:21:27 +03:00
Chris Lu a95929e53c reduce default concurrentWriters to 32 2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision 4c51e6a660 add enableUpsert=true 
and rename config to upsertQuery
 2021-03-30 00:32:03 +03:00
Chris Lu b61e9f6a1a adjust ordering 2021-03-29 00:49:50 -07:00
Konstantin Lebedev 8a95f9c10c iam GetUser 2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision 96c62bd34d scaffold - insertQuery 
Example for upsert query for mysql & postgres variants
 2021-03-29 10:01:12 +03:00
李海 69b2dab9c6 add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id 2021-03-25 18:49:26 +08:00
Konstantin Lebedev 03c7953254 init Iam Api Server 2021-03-25 12:34:11 +05:00
Konstantin Lebedev 9ce82a9cd2 GOMAXPROCS use defaults to the value of runtime.NumCPU 
https://github.com/chrislusf/seaweedfs/issues/1942
 2021-03-24 18:45:58 +05:00
Chris Lu c42b95c596 more help message 
fix https://github.com/chrislusf/seaweedfs/issues/1937
 2021-03-23 17:27:57 -07:00
Chris Lu 85cff10787 mysql: ensure name column is case sensitive 
fix https://github.com/chrislusf/seaweedfs/issues/1931
 2021-03-23 00:46:50 -07:00
Konstantin Lebedev eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
Konstantin Lebedev 06da02739d CheckAllChunkViews() for HEAD requests only 2021-03-16 14:15:17 +05:00
Chris Lu 4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu 35d939a1d2 go fmt 2021-03-14 13:21:02 -07:00
Chris Lu 2a68ddb963 default to empty host to bind to tcp4 and tcp6 2021-03-12 15:32:51 -08:00
Chris Lu 40dc5ac904 mount: add a bit retry when connecting to filer during startup 2021-03-11 16:36:43 -08:00
Chris Lu 105fcd7bfc
Merge pull request #1872 from kmlebedev/grpc_auth 
TLS allowed CommonNames
 2021-03-10 11:17:49 -08:00
Konstantin Lebedev 348e21a08c add comments 2021-03-10 14:42:39 +05:00
Konstantin Lebedev 831953c55c allowed wildcard domain 2021-03-10 14:02:13 +05:00
Konstantin Lebedev 4bf93d6e63 comma-separated 2021-03-10 12:43:13 +05:00
Konstantin Lebedev 190fada1ef TLS allowed commonNames 2021-03-08 21:39:44 +05:00
Konstantin Lebedev c6d3735605 permitCommonNames 
https://github.com/chrislusf/seaweedfs/issues/1841
https://jbrandhorst.com/post/grpc-auth/
 2021-03-08 13:16:17 +05:00
Chris Lu 60686a0bb7 make tcp optional 2021-03-07 14:45:36 -08:00
Chris Lu 1bd880dcdb adds tcp writes benchmark 2021-03-06 14:26:27 -08:00
Chris Lu 400de380f4 volume server: support tcp direct put/get/delete 2021-03-05 02:29:38 -08:00
Chris Lu c5cb346a08 fix compilation error 2021-03-04 12:36:20 -08:00
Chris Lu f5cc96f289 skip checking master peers if not starting master 2021-03-04 12:31:26 -08:00
Chris Lu 40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu c0842fe99f add filer.meta.backup command 2021-03-03 02:02:29 -08:00
Chris Lu 0c5c51eb98 adjust help message 2021-03-02 23:07:39 -08:00
Chris Lu e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu 540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu e52c94640e filer.backup: added to replace filer.replicate 2021-02-28 16:22:27 -08:00
Chris Lu 014a31d11a minor 2021-02-28 16:21:09 -08:00
Chris Lu 678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu 62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
bingoohuang 50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
Chris Lu e1992c83de webdav adds disktype 2021-02-18 15:45:44 -08:00
Chris Lu c2ad6f1047 webdav add replication setting 
fix https://github.com/chrislusf/seaweedfs/issues/1817
 2021-02-18 12:15:09 -08:00
Chris Lu 3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu 6daa932f5c refactoring to get master function, instead of passing master values directly 
this will enable retrying later
 2021-02-17 20:55:55 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
LazyDBA247-Anyvision 7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
Chris Lu 0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision 51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu 4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
bingoohuang 7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
Chris Lu d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu 19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu 9a06c35da4 replicate: incremental sink only contains new and updated files 
address da08402ba2
 2021-01-28 02:39:22 -08:00
Chris Lu da08402ba2 replicate: use creation time for local incremental file sink 
related to https://github.com/chrislusf/seaweedfs/pull/1762
 2021-01-28 02:17:41 -08:00
Chris Lu 822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink 
replication to create time date directory
 2021-01-28 02:03:33 -08:00
Konstantin Lebedev be1062b7fc rename in scaffold 2021-01-28 14:59:20 +05:00
Konstantin Lebedev 02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Chris Lu 5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask 
Use provided umask on mount
 2021-01-28 01:09:20 -08:00
Daniel Santos d38613c49a match upstream 2021-01-27 20:56:41 -07:00
Konstantin Lebedev 6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Chris Lu 0488c84bea adjust help message 2021-01-27 00:17:52 -08:00
Konstantin Lebedev 3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Chris Lu 5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local 
replication to local disk storage
 2021-01-26 22:31:54 -08:00
Daniel Santos 0c64d60fdb Fix log message with correct mode 2021-01-26 21:45:52 -07:00
Daniel Santos 0aaaa9b470 Fix var assignment 2021-01-26 21:06:08 -07:00
Daniel Santos 5a761bb7a4 Enforce umask correctly 2021-01-26 20:42:10 -07:00
Chris Lu ad2a20c8a5 notification add ack and nack 2021-01-26 11:08:44 -08:00
Konstantin Lebedev 612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu 3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu cc1f3907ff filer.sync: default filerProxy to true 2021-01-25 10:14:42 -08:00
Chris Lu 62ebb917e0 mount: avoid the need to specify different cache directory for different mount 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
 2021-01-25 09:10:20 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu 80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu 024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu 84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu 1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu 52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu 96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
LazyDBA247-Anyvision 8eed763b97 better mysql connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of connections.
 2021-01-14 08:14:21 +02:00
Chris Lu 3fb2ed9093 filer.meta.tail: optionally submit metadata changes to ElasticSearch 2021-01-13 03:31:37 -08:00
Chris Lu c5df2577f5 rename 2021-01-13 00:31:19 -08:00
Chris Lu 90c5077610 rename "weed watch" to "weed filer.meta.tail" 2021-01-12 18:48:01 -08:00
Chris Lu 0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu 629c9962e7 filer: add leveldb3 
support dedicated leveldb instance for each bucket
 2021-01-12 02:29:44 -08:00
Chris Lu cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu 2c7148cd55 redis cluster: do not enable route by latency by default 2021-01-11 02:43:53 -08:00